7.66.1

Agent

Prelude

Release on: 2025-06-03

• Please refer to the 7.66.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fixes issue parsing pod list from kubelet when the InPlacePodVerticalScaling feature gate is enabled on the cluster.

Datadog Cluster Agent

Prelude

Released on: 2025-06-03 Pinned to datadog-agent v7.66.1: CHANGELOG.

7.66.0

Agent

Known issue

This version (and previous) of the Datadog Agent is not compatible with Kubernetes 1.33+ versions due to the Feature Gate InPlacePodVerticalScaling that became enabled by default. This flag modifies the kubelet /pods output preventing the correct behaviour of the Datadog Agent. The recommendation is to upgrade to Agent v7.66.1, which is fully compatible with the latest (and previous) Kubernetes versions. More details can be found in this issue.

Prelude

Release on: 2025-05-22

• Please refer to the 7.66.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• If you use a custom Agent username and password on Windows with an Active Directory domain account and you want to remotely upgrade the Agent using Fleet Automation then you must provide the DDAGENTUSER_PASSWORD option when upgrading to 7.66 or later. For more information see the features release notes.

New Features

• Added a new WLAN check that monitors the Wi-Fi interface on the host system. This check is only available for macOS systems.

• Fleet Automation now supports remote upgrades when using a custom Agent username and password on Windows.

  Windows stores the password as an encrypted LSA local private data object that is only accessible to local Administrators. Windows Service Manager stores service account passwords in the same location. For more information, see the Microsoft documentation on Storing Private Data and Private Data Objects.

  Uninstalling the Agent removes the encrypted password from the LSA.

  To avoid providing and manually managing the account password, consider using a Group Managed Service Account (gMSA). For more information, see Installing the Agent with a gMSA account.

• Adds support for persisting of non-core integrations during Agent upgrades on Windows platforms. To disable, set the INSTALL_PYTHON_THIRD_PARTY_DEPS="0" property during the installation of the MSI.

• adds the ability for the Agent to tail logs via the kubelet's API.

• Support multiple authentication methods for a subnet in network devices autodiscovery.

• Use cdpCacheSecondaryMgmtAddr and cdpCacheAddress for CDP topology links in case cdpCachePrimaryMgmtAddr is empty or of an unsupported type.

• Enable language detection via tracers metadata

Enhancement Notes

• Add a build option (--glibc, enabled by default) to build the Agent on glibc environment. On the other libc environments like musl, the Agent should be built with --no-glibc option. The option enables system-probe gpu module and corechecks gpu collector using github.com/NVIDIA/go-nvml which depends on a glibc-extended definition.
• OpenTelemetry instrumentation scope attributes are now converted into span attributes.
• Utilize distributed senders and rtt fairness algorithms to improve logs pipeline throughput and fairness.
• Adds | separator to the DogstatsD debug table (from agent dogstatsd-stats) when not requesting JSON output. This allows the table to render properly in markdown format.
• Get the k8s cluster name from an AKS node label if present
• APM: Improve debug logging for ignore_resources configuration by showing what rule resulted in a trace being ignored.
• Added a new feature flag disable_operation_and_resource_name_logic_v2 in DD_APM_FEATURES that replaces enable_operation_and_resource_name_logic_v2. The new operation name logic for OTLP is now opt-out instead of opt-in.
• Added an option for the Oracle integration to template the database instance identifier.
• The Oracle integration now supports the empty_default_hostname option to omit host from metrics
• Added the exclude_hostname option to the Oracle integration
• The OTLP receiver gRPC server in the trace agent now respects the config otlp_config.receiver.protocols.grpc.max_recv_msg_size_mib or env var DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_MAX_RECV_MSG_SIZE_MIB.
• added new expvar "submission_error_count" on process endpoint to be shown during user status checks
• Reduce the memory footprint of the logs pipeline by eliminating unnecessary fields in the log payload.
• The Windows Agent MSI now sets the Agent account password to the provided DDAGENTUSER_PASSWORD value when it is a local account. Previously, if the provided password did not match the account password, the Agent would fail to start.

Bug Fixes

• Addressed a bug in the cluster-agent API that prevented tag extraction for annotations from working due to client side filtering. The fix was implemented in both the node-agent and the cluster-agent. Now, node-agent clients specify the annotations filter when querying the cluster-agent.
• APM: Fix an issue where apm_config.ignore_resources only removes the root span instead of discarding the whole trace when using OTLP ingestion.
• When using OTLP ingest with metrics, the instrumentation_scope_metadata_as_tags option now outputs the instrumentation_scope tag instead of the deprecated instrumentation_library tag.
• Prevents the index out of range error caused when trying to match inspect layer digests to history layers on some images.
• Fix clusterchecks dispatching on the Cloud Foundry Cluster Agent
• Fixed an issue in the KSM check where, when using pod_collection_mode: node_kubelet, the Agent reported incorrect values for the kubernetes_state.container.status_report.count.waiting metric.
• Fixes a bug that Agent OTLP ingestion fails to start when the config otlp_config.receiver.protocols.grpc.max_recv_msg_size_mib or env var DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_MAX_RECV_MSG_SIZE_MIB is set to a string.
• Process Agent files are added to the flare archive instead of displaying "no session token provided".

Other Notes

• Add a new Agent telemetry tag auth to the API telemetry metrics. This tag is used to evaluate the impact of reworking the authentication system for inter-process communication.
• Add a new metric counter to the Agent telemetry for status rendering errors. This is used to detect potential issues with an ongoing change in the status rendering.

Datadog Cluster Agent

Prelude

Released on: 2025-05-22 Pinned to datadog-agent v7.66.0: CHANGELOG.

New Features

• For workload selection in auto-instrumentation, users can now use the Kubernetes native valueFrom as an alternative to value in ddTraceConfigs. This enables dynamic, user-defined and label based value propagation to the tracing SDKs, like DD_SERVICE.
• Collect EndpointSlice manifests in the orchestrator check.
• Tag resources from Cluster Agent Orchestrator check with all static tags.

Bug Fixes

• Fix major data races in the orchestrator check for the Kubernetes resource collection.
• Fix data race in autodiscovery cluster checks provider.
• Fix data race in autodiscovery kube services provider.
• Fixes an issue with autoinstrumentation where sometimes a DD_SERVICE is not consistent between containers and init containers.
• The auto-instrumentation webhook no longer mutates the istio-proxy container. This fixes an issue with Kubernetes-native sidecars and the istio service mesh where a standard sidecar is moved to be the first init container by istio after it was mutated by auto-instrumentation.
• The cluster-agent kubernetes_metadata API now supports client specified annotations filtering. Clients can pass along filters as query parameters like '?filter=abc&filter=def'.

7.65.2

Agent

Prelude

Release on: 2025-05-13

• Please refer to the 7.65.2 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• (datadog-fips-agent) Ensure the post-install script always rebuilds fipsmodule.cnf in case the embedded OpenSSL is updated.
• The embedded OpenSSL on Windows no longer links against zlib (which wasn't included), preventing errors related to accidentally loading a version of zlib installed on the host.
• On Windows, restarting the datadogagent service now also restarts the Datadog Installer service to ensure configuration changes take effect.

Datadog Cluster Agent

Prelude

Released on: 2025-05-13 Pinned to datadog-agent v7.65.2: CHANGELOG.

Bug Fixes

• Fix wrong computation of the init container resources in the autoinstrumentation webhook.

7.65.1

Agent

Prelude

Release on: 2025-05-08

• Please refer to the 7.65.1 tag on integrations-core for the list of changes on the Core Checks

Datadog Cluster Agent

Prelude

Released on: 2025-05-08 Pinned to datadog-agent v7.65.1: CHANGELOG.

Bug Fixes

• Customers relying on the deprecated v1 implementation of the auto instrumentation webhook will no longer be forced to use the v2 implementation. This will provide additional time for customers to migrate from the v1 to the v2 implementation and ensure the v2 implementation adequately supports all existing use cases.

7.65.0

Agent

Prelude

Release on: 2025-05-06

• Please refer to the 7.65.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Covers more gRPC status code values sent by tracer libraries
• Aggregate APM stats payloads by gRPC code

New Features

• The Agent MSI for Windows now installs the .NET APM Instrumentation libraries and automatically configures APM Instrumentation for IIS. To enable this feature, specify the following properties in the MSI command line:

  DD_APM_INSTRUMENTATION_ENABLED=iis DD_APM_INSTRUMENTATION_LIBRARIES=dotnet:3

  Requires IIS 10.0 or later. For more information see [Single Step APM Instrumentation](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/single-step-apm/).

• APM: Added new OpenLineage event proxy endpoint.

• A new implementation of auto multi-line detection for logs is now available. This new implementation is more flexible and powerful than the previous one. It now supports arbitrary timestamps, continuous matching, and more flexible configuration. It will be enabled automatically when logs_config.auto_multi_line_detection is set to true. You can opt out of the new implementation by setting logs_config.force_auto_multi_line_detection_v1 to true.

• APM: Container-based primary tags are now available by default.

Enhancement Notes

• Add a new host tag called orch_cluster_id that is set to the Kubernetes cluster ID generated by the cluster-agent. This is added only for the hosts that are part of a Kubernetes cluster. This tag is also used on every cluster-level resources like kubernetes_state.* metrics, kubernetes events, and orchestrator level resources.

• Add admission.datadoghq.com/apm-inject.debug: "true" annotation to inject DD_APM_INSTRUMENTATION_DEBUG=true, DD_TRACE_STARTUP_LOGS=true and DD_TRACE_DEBUG=true to an application container. This is useful for debugging Single Step Instrumentation or tracer issues in Kubernetes environments.

• Add metrics origins for Sonatype Nexus, Silverstripe CMS and Anecdote integrations.

• Add the propagate_agent_tags setting. When set to true, the tags from the Agent host are added to the check's tag for all instances.

• Agents are now built with Go 1.23.7.

• Agents are now built with Go 1.23.8.

• Allow integrations to self-declare as HA Supported.

• Improved trace context creation from Step Function execution context:

  • Now utilizes State.RetryCount and Execution.RedriveCount to generate parent IDs deterministically, preventing collisions with retry spans.
  • Supports multi-level trace merging, allowing an arbitrary number of Lambda and Step Function traces to be linked together while maintaining root context.

  This update brings feature parity with Node and Python Lambda layers in the Universal runtimes.

• The hostname subcommand for the Agent now tries to reach out to the running Agent, and falls back to computing the hostname locally if the Agent is not running. You can use the --local option to force the local computation.

• Cisco SD-WAN: improve memory usage.

• Adds a metadata_ip_resolution_from_hostname config option to use hostname DNS resolution as the preferred method to compute the host IP address

• Metrics are now sent over HTTP/2 when possible. A slight change to the connection handling to take full advantage of this means multiple requests can be in flight at any one time. This can be configured with the forwarder_max_concurrent_requests option, default is 10.

• Added configurations network_path.collector.source_excludes and network_path.collector.dest_excludes, which allow ignoring CIDR ranges in the Network Path collector.

• Added support for querying the pod list through the API server. This feature, enabled by setting kubeletUseApiServer to true, allows the Agent to retrieve pod metadata directly from the API server instead of the kubelet. This resolves issues when direct access to the kubelet /pods endpoint is restricted. The agent will continue to query the kubelet for /metrics and /stats/summary. Must add pods to Agent cluster role.

• Process checks now run in the core Agent by default on Linux. Process Agent will only run if needed for other configured features.

• Include container_instance_arn in ECS Task payloads.

• Include APIVersion and Kind to Kubernetes manifests.

• The Python runtime is now loaded when the first check is loaded, rather than when the agent starts. This will eventually avoid having to load Python when no Python check is enabled. This change can be reverted by setting python_lazy_loading: false in your configuration.

• When the API key can't be accessed in serverless, it is now logged as an error instead of debug.

• Added FIPS compliance support for Lambda Extension GovCloud customers. The Lambda Extension now uses FIPS-enabled endpoints for AWS KMS and Secrets Manager when running in GovCloud regions.

• APM: Return the correct content-type header when rejecting trace payloads.

• APM: Expose an "obfuscation_version" value via the /info endpoint. Accept a new header "Datadog-Obfuscation-Version" for incoming stats payloads; if any non-zero value is set, the trace-agent will not attempt to obfuscate these payloads as they have already been obfuscated by the tracer.

• Adds functionality to construct a DD span from datadog. attributes on an incoming span. By default, it checks for a span attribute starting with datadog.; if present, it uses this to compute the corresponding Datadog field (e.g., datadog.service is used to set ddspan.Service). This will override other sources for the same field (e.g., if both datadog.service and service.name are present, datadog.service is used). By default, if a field is missing, it will be recomputed (e.g., if there's no datadog.env, it will look for env in deployment.environment). However, if the config option otlp_config.traces.ignore_missing_datadog_fields is specified, the field will not be recomputed. This option allows users to explicitly specify blank fields if they desire (e.g., set datadog.env = "").

  The following functionality is removed: if both http.response.status_code and http.status_code were present in the span attributes, the former was preferred. Similarly, http.request.method was preferred over http.method. In this release, the key encountered first is the one used.

Security Notes

• The OPENSSL_CONF and OPENSSL_MODULES environment variables are now set when running the datadog-fips-agent in FIPS mode. This allows applications to use the OpenSSL configuration and modules configured with the Agent installation.

Bug Fixes

• Fixes bugs in the analyze-logs subcommand, and validates the logs config for invalid syntax.
• Fixes a race condition introduced in PR #33521 around the PythonHome getting set in an init hook.
• Fix the NVIDIA Jetson check to support parsing the tegrastats output when there are multiple Graphics Processing Clusters (GPCs). In that case, the metric nvidia.jetson.gpu.freq is emitted for each GPC and tagged with gpc:0, gpc:1, and so on.
• Fixes issue where the CRI-O image status endpoint would be queried when container_image.enabled was set to false.
• CWS: Fixed a bug that prevented more than one runtime security rule from being disabled, and caused incorrect policy information to be reported in ruleset loaded events.
• Fixed a spinlock corruption in the Windows driver ddprocmon.sys that can cause system hangs and memory corruption when system-probe.exe is shutting down.
• Fixes an issue where the system-probe would spuriously log the error netlink conntracker requires NET_ADMIN capability on the Fargate CNM preview.
• Cisco SD-WAN: fix an issue that could lead to incorrect metrics tagging.
• Fix the shebang of the gstatus binary to use the embedded Python environment.
• APM: fix issue where a new span field for span events wasn't properly serialized when received in MessagePack format.
• Environment variables that the kubelet collector receives and are not explicitly defined in plaintext will now be ignored.

Other Notes

• PHP support is enabled by default in the Cluster Agent
• Metrics Origin entries for Infiniband and Celery integrations
• Update OpenSSL from 3.4.0 to 3.4.1.
• PHP support for Single Step Instrumentation is now installed by default.
• Add metric origin for the Velero integration.
• Adds the Datadog Installer service to the Agent MSI. The Datadog Installer service will start on boot but will exit unless the Remote Upgrades pr...

7.64.3

Agent

Prelude

Release on: 2025-04-10

Enhancement Notes

• Agents are now built with Go 1.23.8.

Bug Fixes

• Add a configuration option, enable_nvml_detection, disabled by default, to stop the Agent from trying to find the NVML library on startup.
• Fix remote tagger initialization log message to be logged only once.

Datadog Cluster Agent

Prelude

Released on: 2025-04-10 Pinned to datadog-agent v7.64.3: CHANGELOG.

7.64.2

Agent

Prelude

Release on: 2025-04-02

• Please refer to the 7.64.2 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Process Agent files are added to the flare archive instead of displaying "no session token provided".

Datadog Cluster Agent

Prelude

Released on: 2025-04-02 Pinned to datadog-agent v7.64.2: CHANGELOG.

7.64.1

Agent

Prelude

Release on: 2025-03-20

• Please refer to the 7.64.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fixed Python package dependency issue preventing integrations from loading on Windows

Datadog Cluster Agent

Prelude

Released on: 2025-03-20 Pinned to datadog-agent v7.64.1: CHANGELOG

7.64.0

Agent

Known Issues

This version contains a Python package dependency (cryptography) issue that can prevent Python integrations from loading on Windows. A workaround is to set the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 at the machine level and restart the Agent. If the issue persists, the recommendation at this time is to downgrade to Agent v7.63.3 or upgrade to v7.64.1 when it becomes available.

Prelude

Release on: 2025-03-19

• Please refer to the 7.64.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Add the new following docker images, containing every components of the Datadog Agent, including the new Otel-Agent:
  • 7.X.Y-full
  • 7-full
  • latest-full
• The datadog-fips-agent now builds with the 'requirefips' buildtag in the 7.64 Linux release and no longer needs the GOFIPS=1 environment variable set to enable FIPS mode. When upgrading from 7.63, this can be removed.
• Enables IMDSv2 by default on all EC2 instance hosts by updating theec2_imdsv2_transition_payload_enabled flag from false to true. If IMDSv2 hasn’t been explicitly enabled and the hostname isn’t set to the instance ID, the display name may change to the instance ID without affecting Agent behavior. For more information, see the IMDSv2 Enablement by Default documentation.
• Bump the Python version to 3.12.9
• ECS task collection is now enabled by default (see ecs_task_collection_enabled in the datadog.yaml configuration).

New Features

• Added support for obfuscation for valkey command. This feature is enabled by default. To disable it, set DD_APM_OBFUSCATION_VALKEY_ENABLED=false. To replace all valkey command arguments with a single ?, set DD_APM_OBFUSCATION_VALKEY_REMOVE_ALL_ARGS=true (default: false).
• APM: Add support for multi-region failover. This feature is controlled via Remote Config and allows the trace-agent to switch to a failover data center when enabled.
• Add new card: common field to DogStatsD Datagram specification to allow customer to specify the cardinality of the metric. This field is optional.
• Log Agent now officially supports http2 transport to proxy.

Enhancement Notes

• Add per-process GPU tagging when running process checks on the Core Agent.
• Enable pymem.inuse metric when Agent telemetry is enabled
• In rare cases, when the Agent's Network Performance Monitor is enabled and the Agent is identified as contributing to a "Blue Screen of Death" (BSOD) event, Agent telemetry is used to generate a payload that includes Agent's driver code offset for further analysis.
• Adds a flag for setting the inactivity timeout and adds support for "check name" for the analyze logs subcommand
• Agents are now built with Go 1.23.6.
• Added a new feature flag disable_receive_resource_spans_v2 in DD_APM_FEATURES that replaces enable_receive_resource_spans_v2 - the refactored implementation of ReceiveResourceSpans for OTLP is now opt-out instead of opt-in.
• Enable IMDSv2 by default for all EC2 instance hosts if IMDSv2 usage was not explicitly enabled.
• Added capability to generate profiling data in a flare requested via Remote Config
• Image layer digests will no longer report as "<missing>" from Docker runtimes.
• Upgraded github.com/DataDog/go-sqllexer to v0.1.1, resulting in reduced CPU usage and improved memory allocation efficiency.
• Upgraded github.com/DataDog/go-sqllexer to v0.1.3 to fix a bug that caused the lexer to panic when trimming identifier quotes.
• Added a new language detector that uses the apm-inject propagated language, if available.
• The KSM core check is now capable of retrying its coordination with the Kubernetes API Server. Failed queries in the check configuration will no longer block the check from being scheduled.
• • Add logging for log compression configuration.
  • The Agent now validates the log compression setting and falls back to the default compression if an invalid option is provided.
• Report resource requirements as native sidecars when restartPolicy=Always is used.
• Improved the behavior of the SQL obfuscator cache to allow caching of both obfuscated and normalized queries
• APM: Fix a formatting bug where the trace-agent's PID from "agent status" could be displayed in scientific notation for large PIDs.

Deprecation Notes

• logs.processed and logs.sent metrics are no longer emitted by the Agent
• Deprecation warnings added for the fips-proxy configuration keys (e.g. fips.*) as this is planned to be unsupported in 7.65+ releases of the datadog-agent. Please use the datadog-fips-agent for FIPS compliance instead.
• APM: The existing /config/set endpoint on the trace-agent is now deprecated in favor of the /config/set endpoint on the debug port on the trace-agent (default: 5012). The old endpoint will be removed in a future version.

Bug Fixes

• Fixed parsing of group resource strings to support groups with periods.
• Fixed a bug in the DogStatsD Unix socket server that caused metrics to miss container tags and the Agent to report matched PID for the process is 0 warnings.
• Fix issue with FIPS image where some build tags were missing for the process and trace agents during packaging.
• Makes CWS report the correct container ID on EKS Fargate.
• Fix an issue where ingestion_reason:probabilistic is set even when an OTLP span was sampled by the Error Sampler. To enable the Error Sampler for OTLP spans, you need to set DD_OTLP_CONFIG_TRACES_PROBABILISTIC_SAMPLER_SAMPLING_PERCENTAGE to 99 or lower, or enable DD_APM_PROBABILISTIC_SAMPLER_ENABLED and set DD_APM_PROBABILISTIC_SAMPLER_PERCENTAGE to 99 or lower.
• version-manifest.json and version-manifest.txt files now correctly reflect the packages content.
• Prevent journald and windows event logs from being errantly marked as truncated in specific circumstances.
• Obfuscation Cache Size Calculation: Resolved an issue where the cache item size was underestimated by not accounting for the Go struct overhead (including struct fields and headers for strings and slices). This fix ensures a more accurate calculation of cache item memory usage, leading to better memory efficiency and preventing over-allocation of NumCounters in cache configurations.
• Fix potential panic in journald and Windows event tailers during system shutdown
• Remove leading expressions in parentheses during SQL normalization.
• APM: Fix a rare panic that can occur when using client side stats in the tracers.
• APM: Fix an issue where the environment tag was normalized incorrectly. This resulted in some valid envs, like 123foo, having the leading digits removed. This fix allows these envs to pass through unedited.

Other Notes

• Add multi line log aggregation telemetry.

Datadog Cluster Agent

Prelude

Released on: 2025-03-19 Pinned to datadog-agent v7.64.0: CHANGELOG.

Upgrade Notes

• Datadog Autoscaling is upgraded to use DatadogPodAutoscaler CRD v1alpha2 instead of v1alpha1. Remote (created in Datadog) autoscalers are automatically migrated. In-cluster (Local) autoscalers need to be migrated manually.

New Features

• Enable collection of Pod Disruption Budgets by default in the orchestrator check.

• Target-based workload selection is now available for Single Step Instrumentation. This feature enables you to instrument specific workloads using pod and namespace label selectors. By applying user-defined labels, you can select workloads for instrumentation without modifying applications. For example, the following configuration injects the Python tracer with a default version for pods labeled with language=python:

instrumentation:
  enabled: true
  targets:
    - name: "Python Services"
      podSelector:
        matchLabels:
          language: "python"
      ddTraceVersions:
        python: "default"

• Targets can also be chained together, with the first matching rule taking precedence. For example, the following configuration installs the Python tracer for pods labeled language=python and the Java tracer for pods in a namespace labeled language=java. If a pod matches both rules, the first match takes precedence:

instrumentation:
  enabled: true
  targets:
    - name: "Python Services"
      podSelector:
        matchLabels:
          language: "python"
      ddTraceVersions:
        python: "default"
    - name: "Java Namespaces"
      namespaceSelector:
        matchLabels:
          language: "java"
      ddTraceVersions:
        python: "default"

• Targets support tracer configuration options in the form of environment variables. All options must have the DD_ prefix. The following example installs the Python tracer with profiling and data jobs enabled:

instrumentation:
  enabled: true
  targets:
    - name: "Python Apps"
      podSelector:
        matchLabels:
          l...

7.63.3

Agent

Prelude

Release on: 2025-03-04

• Please refer to the 7.63.3 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fully disable the X25519Kyber768Draft00 key exchange mechanism to avoid issues with firewalls not supporting multi-packet key exchanges, in particular AWS Network Firewall and Suricata.

Datadog Cluster Agent

Prelude

Released on: 2025-03-04 Pinned to datadog-agent v7.63.3: CHANGELOG.