-
Notifications
You must be signed in to change notification settings - Fork 325
Open
Labels
type: bugBug report and fixBug report and fix
Description
Tracer Version(s)
1.54.0, 1.55.0
Java Version(s)
25.0.36
JVM Vendor
Eclipse Adoptium / Temurin
Bug Report
After updating from 1.40.1 to 1.55.0, I noticed a lot of requests for a Resource named OperationHandler.handle inside APM on a Service. I eventually linked it to Spring Actuator requests, mostly from a Kubernetes health check.
We have ".*/actuator.*" set to be excluded at the Datadog Agent level in the cluster, but there seemed to be a new span created for each actuator call.
I gradually reverted the library version and once I got to 1.53.0, the issue went away.
Here's a bit of an excerpt for a single Trace from the logs with DD_TRACE_DEBUG=true:
2025-11-07 14:43:01.907 [tomcat-handler-60] DEBUG datadog.trace.agent.core.propagation.HttpCodec - Extract incomplete context TagContext{baggage={}, }
2025-11-07 14:43:01.907 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `null`->`null` with priority 0->0 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.908 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpan - Started span: DDSpan [ t_id=3536004845519906260, s_id=8720177333644726388, p_id=0 ] trace=edi-gateway-api/servlet.request/servlet.request tags={_dd.injection.mode=k8s, _dd.profiling.enabled=0, _dd.trace_span_attribute_schema=0, env=dev, language=jvm, process_id=1, runtime-id=297c889b-aaf3-4e12-a504-b8eb641f364b, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=0, forceKeep=false, links=[]
2025-11-07 14:43:01.908 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `null`->`GET /edi/api/actuator/health` with priority 0->1 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.909 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `GET /edi/api/actuator/health`->`GET /edi/api/actuator/health` with priority 1->1 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.910 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `GET /edi/api/actuator/health`->`GET /edi/api/actuator/health` with priority 1->3 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.913 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `GET /edi/api/actuator/health`->`GET /edi/api/actuator/health` with priority 3->3 for traceId=3536004845519906260 spanId=8720177333644726388
2025-11-07 14:43:01.916 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `null`->`null` with priority 0->0 for traceId=3536004845519906260 spanId=1749850157596654480
2025-11-07 14:43:01.917 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpan - Started span: DDSpan [ t_id=3536004845519906260, s_id=1749850157596654480, p_id=8720177333644726388 ] trace=edi-gateway-api/spring.handler/spring.handler tags={_dd.injection.mode=k8s, env=dev, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=0, forceKeep=false, links=[]
2025-11-07 14:43:01.918 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpanContext - setResourceName `null`->`OperationHandler.handle` with priority 0->0 for traceId=3536004845519906260 spanId=1749850157596654480
2025-11-07 14:43:01.921 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpan - Finished span (PENDING): DDSpan [ t_id=3536004845519906260, s_id=1749850157596654480, p_id=8720177333644726388 ] trace=edi-gateway-api/spring.handler/OperationHandler.handle *measured* tags={_dd.injection.mode=k8s, component=spring-web-controller, env=dev, language=jvm, span.kind=server, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=4889105, forceKeep=false, links=[]
2025-11-07 14:43:01.924 [tomcat-handler-60] DEBUG datadog.trace.agent.common.writer.RemoteWriter - Enqueued for serialization: [DDSpan [ t_id=3536004845519906260, s_id=8720177333644726388, p_id=0 ] trace=edi-gateway-api/servlet.request/GET /edi/api/actuator/health *measured* tags={_dd.agent_psr=1.0, _dd.injection.mode=k8s, _dd.profiling.enabled=0, _dd.trace_span_attribute_schema=0, _sample_rate=1, component=tomcat-server, env=dev, http.hostname=100.64.82.138, http.method=GET, http.route=/edi/api/actuator/health, http.status_code=200, http.url=http://100.64.82.138:8088/edi/api/actuator/health, http.useragent=ELB-HealthChecker/2.0, language=jvm, peer.ipv4=172.18.31.109, peer.port=16282, process_id=1, runtime-id=297c889b-aaf3-4e12-a504-b8eb641f364b, servlet.context=/, servlet.path=/edi/api/actuator/health, span.kind=server, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=15676391, forceKeep=false, links=[], DDSpan [ t_id=3536004845519906260, s_id=1749850157596654480, p_id=8720177333644726388 ] trace=edi-gateway-api/spring.handler/OperationHandler.handle *measured* tags={_dd.injection.mode=k8s, component=spring-web-controller, env=dev, language=jvm, span.kind=server, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=4889105, forceKeep=false, links=[]]
2025-11-07 14:43:01.924 [tomcat-handler-60] DEBUG datadog.trace.agent.core.DDSpan - Finished span (WRITTEN): DDSpan [ t_id=3536004845519906260, s_id=8720177333644726388, p_id=0 ] trace=edi-gateway-api/servlet.request/GET /edi/api/actuator/health *measured* tags={_dd.agent_psr=1.0, _dd.injection.mode=k8s, _dd.profiling.enabled=0, _dd.trace_span_attribute_schema=0, _sample_rate=1, component=tomcat-server, env=dev, http.hostname=100.64.82.138, http.method=GET, http.route=/edi/api/actuator/health, http.status_code=200, http.url=http://100.64.82.138:8088/edi/api/actuator/health, http.useragent=ELB-HealthChecker/2.0, language=jvm, peer.ipv4=172.18.31.109, peer.port=16282, process_id=1, runtime-id=297c889b-aaf3-4e12-a504-b8eb641f364b, servlet.context=/, servlet.path=/edi/api/actuator/health, span.kind=server, thread.id=137, thread.name=tomcat-handler-60, version=2.36.0-feature_java_25-SNAPSHOT-2800}, duration_ns=15676391, forceKeep=false, links=[]
2025-11-07 14:43:01.996 [dd-remote-config] DEBUG datadog.communication.ddagent.DDAgentFeaturesDiscovery - discovered traceEndpoint=v0.4/traces, metricsEndpoint=v0.6/stats, supportsDropping=true, supportsLongRunning=true, dataStreamsEndpoint=v0.1/pipeline_stats, configEndpoint=null, evpProxyEndpoint=evp_proxy/v4/, telemetryProxyEndpoint=telemetry/proxy/
2025-11-07 14:43:02.942 [dd-trace-processor] DEBUG datadog.trace.agent.common.writer.ddagent.DDAgentApi - Success while sending 1 (size=1KB) traces. Total: 89, Received: 89, Sent: 89, Failed: 0.
2025-11-07 14:43:02.943 [dd-trace-processor] DEBUG datadog.trace.agent.common.writer.PayloadDispatcherImpl - Successfully sent 1 traces to the API
Expected Behavior
No extra Resources in the APM Service page relating to ignored Resources
Reproduction Code
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
type: bugBug report and fixBug report and fix