DSMON-1102: Add configuration-level dynamic tags for JMX attribute values #581

piochelepiotr · 2025-10-15T21:10:52Z

Problem

Kafka exposes its cluster ID via JMX, but as an attribute value rather than in the bean path:

Bean: kafka.server:type=KafkaServer,name=ClusterId
Attribute: Value → returns the actual cluster ID (e.g., "kafka-prod-cluster-01")

Currently, there's no way to extract this value and tag all Kafka metrics with it. Users must either:

Hardcode the cluster ID for each environment (error-prone, not maintainable)
Don't use cluster_id tag: End up mixing metrics for topics with the same name, in different clusters. Can't build reliable features on top of these metrics (For Data Streams Monitoring)

This becomes particularly problematic when monitoring multiple Kafka clusters or across different environments (dev/staging/prod).

Solution

This PR adds support for configuration-level dynamic tags that can extract values from JMX bean attributes at runtime and apply them to all matching metrics.

Syntax:

instances:
  - host: kafka-broker
    port: 9999
    conf:
      - include:
          domain: kafka.server
          attribute:
            MessagesInPerSec:
              metric_type: rate
         dynamic_tags:
           - tag_name: kafka_cluster_id
              bean_name: kafka.server:type=KafkaServer,name=ClusterId
              attribute: Value

Example: Kafka Cluster ID

All Kafka metrics automatically tagged with the cluster ID:

kafka.messages_in[cluster_id:kafka-prod-cluster-01,...] = 1500 msg/s
kafka.bytes_in[cluster_id:kafka-prod-cluster-01,...] = 150MB/s

piochelepiotr · 2025-10-16T02:41:47Z

src/main/java/org/datadog/jmxfetch/Filter.java

+            String tagName = entry.getKey();
+            String tagValue = entry.getValue();
+
+            if (tagValue != null && tagValue.contains("#") && tagValue.startsWith("$")) {


I want to make sure normal tags are not being mistaken for JMX tags, but I think this condition is strict enough that it won't happen. A tag starting with $ is not valid anyway.
Also, the parsing is in a try / catch, the catch adds it to normal tags.

I want to make sure normal tags are not being mistaken for JMX tags,

More robust way would be to have a distinct configuration section for computed tags, and would avoid the need for bespoke syntax.

Also, jmxfetch integrations are configured in yaml, where "#" denotes a comment. Would it be possible to use a different separator? Or perhaps even be explicit and use two keys: bean_name and attribute.

Makes sense! You actually made me think about backward compatibility (I want to make sure users can update their integration configurations without worrying about breaking their integration).

See PR: DataDog/integrations-core#21687

Adding a dynamic_tags section at the same level as the include section is backwards compatible (I can't add the dynamic_tags section inside the include section). I find the current format pretty clean, and backwards compatible.

vickenty · 2025-10-16T09:36:23Z

src/main/java/org/datadog/jmxfetch/Filter.java

+            String tagName = entry.getKey();
+            String tagValue = entry.getValue();
+
+            if (tagValue != null && tagValue.contains("#") && tagValue.startsWith("$")) {


I want to make sure normal tags are not being mistaken for JMX tags,

More robust way would be to have a distinct configuration section for computed tags, and would avoid the need for bespoke syntax.

Also, jmxfetch integrations are configured in yaml, where "#" denotes a comment. Would it be possible to use a different separator? Or perhaps even be explicit and use two keys: bean_name and attribute.

src/main/java/org/datadog/jmxfetch/DynamicTag.java

vickenty · 2025-10-16T09:49:16Z

src/main/java/org/datadog/jmxfetch/Configuration.java

        return getInclude() != null && !getInclude().isEmptyFilter();
    }
+
+    /** Resolve dynamic tags for this configuration using cached values. */


Could you please make the comment more explicit? What does "Resolve" mean in this case, what are the inputs and outputs?

vickenty · 2025-10-17T11:13:56Z

src/main/java/org/datadog/jmxfetch/DynamicTag.java

+        Object attrObj = config.get("attribute");
+
+        if (tagNameObj == null || beanObj == null || attrObj == null) {
+            log.warn("Invalid dynamic tag config: missing 'tag_name', 'bean' or 'attribute' key"); 


Let's be explicit what is missing.

vickenty · 2025-10-17T11:15:45Z

src/main/java/org/datadog/jmxfetch/DynamicTag.java

+
+        Map<String, Object> config = (Map<String, Object>) tagConfig;
+        Object tagNameObj = config.get("tag_name");
+        Object beanObj = config.get("bean");


I may be wrong, but I think this refers to the same string as bean_name in the filter section. Would it make sense to use the same key name here?

Suggested change

Object beanObj = config.get("bean");

Object beanObj = config.get("bean_name");

vickenty · 2025-10-17T11:20:16Z

src/main/java/org/datadog/jmxfetch/Configuration.java

    private Filter include;
    private Filter exclude;
+    private List<DynamicTag> dynamicTags = null;
+    private Map<String, String> resolvedDynamicTags = null;


Up until now, this object represented only user configuration with no runtime data. Would it be possible to keep it this way?

We can keep any runtime data inside Instance and JmxAttribute. For example JmxAttribute can retrieve tag values from the instance cache once we found a matching configuration.

vickenty · 2025-10-22T13:14:22Z

src/main/java/org/datadog/jmxfetch/JmxAttribute.java

    /** Sets a matching configuration for the attribute. */
-    public void setMatchingConf(Configuration matchingConf) {
+    public void setMatchingConf(Configuration matchingConf, 
+            Map<String, String> resolvedDynamicTags) {


Let's use a separate setter for dynamic tags.

vickenty · 2025-10-22T13:20:54Z

src/main/java/org/datadog/jmxfetch/Instance.java

+        }
+
+        for (DynamicTag dynamicTag : allDynamicTags) {
+            String cacheKey = dynamicTag.getBeanName() + "#" + dynamicTag.getAttributeName();


This is duplicated here and in the next method. Would it be possible to make it a method of DynamicTag?

vickenty · 2025-10-22T13:22:24Z

src/main/java/org/datadog/jmxfetch/Instance.java

+            if (!this.dynamicTagsCache.containsKey(cacheKey)) {
+                Map.Entry<String, String> resolved = dynamicTag.resolve(connection);
+                if (resolved != null) {
+                    this.dynamicTagsCache.put(cacheKey, resolved);


If dynamic tag references a non-existing attribute, we would try to retrieve it multiple times (once for each config it is part of). Would it be possible to cache negative lookups as well, so we don't spam the logs with identical errors?

src/test/resources/jmx_config_dynamic_tags_multi.yaml

piochelepiotr · 2025-10-24T15:59:04Z

/merge

dd-devflow-routing-codex · 2025-10-24T15:59:08Z

View all feedbacks in Devflow UI.

2025-10-24 15:59:08 UTC ℹ️ Start processing command /merge

2025-10-24 15:59:13 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 0s (p90).

2025-10-24 16:13:35 UTC ℹ️ MergeQueue: This merge request was merged

Add dynamic tags support to JMX Fetch

b91a371

piochelepiotr force-pushed the piotr.wolski/add-dynamic-tags branch from 6a260b9 to 882f7c8 Compare October 15, 2025 21:11

Use tags from JMX sections, not instance tags

9961226

piochelepiotr force-pushed the piotr.wolski/add-dynamic-tags branch from 882f7c8 to 9961226 Compare October 15, 2025 21:17

piochelepiotr commented Oct 16, 2025

View reviewed changes

Use resources files for tests

2c6b688

piochelepiotr marked this pull request as ready for review October 16, 2025 02:50

piochelepiotr requested a review from a team as a code owner October 16, 2025 02:50

vickenty reviewed Oct 16, 2025

View reviewed changes

address PR comments

2ed6089

piochelepiotr requested a review from vickenty October 16, 2025 17:15

piochelepiotr mentioned this pull request Oct 16, 2025

DSMON-1102: Add kafka_cluster_id tag to Kafka cluster metrics DataDog/integrations-core#21687

Merged

3 tasks

piochelepiotr added 4 commits October 16, 2025 21:31

Move dynamic tags to same level as include

19f1c56

remove unused functions

67cc21f

update when tags are resolved

4ee9eb4

separate adding dynamic tags & additional tags

55eb6a7

vickenty reviewed Oct 17, 2025

View reviewed changes

address PR comments

d6e7660

piochelepiotr requested a review from vickenty October 17, 2025 14:56

vickenty reviewed Oct 22, 2025

View reviewed changes

add negative cache and improve code readability

0971987

piochelepiotr requested a review from vickenty October 22, 2025 14:34

vickenty reviewed Oct 24, 2025

View reviewed changes

src/test/resources/jmx_config_dynamic_tags_multi.yaml Show resolved Hide resolved

test invalid dynamic tags & multiple dynamic tags

91ffa17

vickenty approved these changes Oct 24, 2025

View reviewed changes

dd-devflow bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: queued labels Oct 24, 2025

dd-mergequeue bot merged commit efd946a into master Oct 24, 2025
19 checks passed

dd-devflow bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Oct 24, 2025

jszwedko deleted the piotr.wolski/add-dynamic-tags branch October 27, 2025 16:38

	Object beanObj = config.get("bean");
	Object beanObj = config.get("bean_name");

DSMON-1102: Add configuration-level dynamic tags for JMX attribute values #581

DSMON-1102: Add configuration-level dynamic tags for JMX attribute values #581

Uh oh!

Conversation

piochelepiotr commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Example: Kafka Cluster ID

Uh oh!

piochelepiotr Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

piochelepiotr commented Oct 24, 2025

Uh oh!

dd-devflow-routing-codex bot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

piochelepiotr commented Oct 15, 2025 •

edited

Loading

piochelepiotr Oct 16, 2025 •

edited

Loading

dd-devflow-routing-codex bot commented Oct 24, 2025 •

edited

Loading