[BUG] Datadog Agent Network Monitoring prevents AWS VPC CNI interface cleanup in EKS - causes pod IPv4 connectivity failures

[alexandrucojocaru-creatopy]

Problem Description

The Datadog Agent with network monitoring enabled prevents proper cleanup of network interfaces (veth pairs) when pods are deleted in AWS EKS clusters, leading to massive interface accumulation and intermittent IPv4 connectivity failures for pods.

Environment Details

• Platform: Amazon EKS
• Kubernetes Version: 1.33
• Node OS: Bottlerocket OS 1.42.0 (aws-k8s-1.33)
• Cluster IP address family: IPv6
• Instance Type: c7i.4xlarge
• AWS VPC CNI Version: v1.19.6-eksbuild.1
• Datadog Agent Version: 7.70.2
• Datadog Agent Configuration: Network monitoring enabled

Technical Symptoms Discovered

Expected Behavior:

• Node with 35 active pods should have ~35-70 veth interfaces (one pair per pod)
• When pods are deleted, AWS VPC CNI should clean up corresponding veth interfaces

Actual Behavior:

• Node with 35 active pods accumulated 2000+ veth interfaces
• Stale veth interfaces persist after pod deletion

Connectivity Impact:

• Intermittent IPv4 connectivity failures for new pods
• Pods cannot reach external services (DNS resolution works, but TCP connections fail)
• IPv6 connectivity remains unaffected

# Healthy node (without issue):
$ ip addr show | grep ": veth" | wc -l
2  # Expected: matches number of active pods

# Affected node (with Datadog network monitoring):
$ ip addr show | grep ": veth" | wc -l  
2000+  # Problem: massive interface accumulation

1. Without Datadog network monitoring: CNI cleanup works properly
2. With Datadog network monitoring enabled: veth interfaces accumulate indefinitely
3. Hypothesis: Network monitoring hooks prevent AWS VPC CNI from properly cleaning up network interfaces during pod deletion

Steps to Reproduce

1. Deploy AWS EKS cluster with Bottlerocket nodes
2. Install Datadog Agent with network monitoring enabled
3. Deploy and delete pods repeatedly over several days
4. Monitor interface count: ip addr show | grep ": veth" | wc -l
5. Observe interface count growing and not decreasing when pods are deleted

Disabling Datadog network monitoring immediately resolved the issue: the unused network interfaces were deleted after the agent was terminated.

Please let me know what additional diagnostic information would be helpful for investigating this issue.

[fabbing]

Hi Alexandru,

Thanks for reaching out about this issue!
Could you please specify which network monitoring component you are using? Providing an extract of the configuration file containing the network options could help us diagnose the issue.

[alexandrucojocaru-creatopy]

@fabbing Hi Fabrice,

We are using this configuration in the Helm chart:

networkMonitoring:
  enabled: true

As described here.

Setting enabled: false fixes the issue.

[fabbing]

You may have stumbled upon an issue with Cloud Network Monitoring! It would be helpful if we could investigate the issue in more detail.
Would you mind sending us a flare via the Datadog website (https://docs.datadoghq.com/agent/troubleshooting/send_a_flare/?tab=agent#send-a-flare-from-the-datadog-site)? The flare will contain valuable information to help us reproduce and investigate the issue.

[alexandrucojocaru-creatopy]

I sent the flare now, but we have network monitoring disabled now. It's not clear if we should have enabled it before sending.

[fabbing]

Thank you, we will look into this!

[iuliancmarcu]

Hi @fabbing,

Thank you so much for investigating this issue. Tonight, we encountered the same issue even with the networkMonitoring disabled. Exactly the same behaviour - network adapters (ip addr) are not released until we restart the datadog-agent, otherwise our pods lose ipv4 connection after pods are created again and again over the day.

Here is the Helm configuration we use to deploy Datadog in our cluster:

# Datadog Agent Configuration for Development Environment
# Helm Chart: https://github.com/DataDog/helm-charts/tree/main/charts/datadog

# Use public AWS ECR registry for Datadog images
registry: public.ecr.aws/datadog
targetSystem: 'linux'

# ============================================================================
# CLUSTER AGENT CONFIGURATION
# ============================================================================
# The Cluster Agent handles cluster-level operations and reduces API server load
# by centralizing cluster-level queries (events, service discovery, etc.)
clusterAgent:
    enabled: true
    # 2 replicas for high availability in development
    replicas: 2
    
    # Pod Disruption Budget ensures at least 1 pod remains healthy during rollouts
    # This prevents downtime during updates/node maintenance
    createPodDisruptionBudget: true
    pdb:
        create: true
        minAvailable: 1  # Keep at least 1 healthy pod during updates
    
    # Metrics Provider enables custom metrics for Horizontal Pod Autoscaling (HPA)
    metricsProvider:
        enabled: true
    
    # RBAC permissions for cluster-level operations
    rbac:
        create: true
    
    # Admission Controller automatically injects APM configuration into pods
    admissionController:
        enabled: true
        mutateUnlabelled: true  # Auto-inject APM config without requiring pod labels
    
    # High priority ensures cluster agent pods are scheduled first
    priorityClassName: system-cluster-critical
    
    # Resource limits prevent cluster agent from consuming excessive resources
    resources:
        requests:
            cpu: 200m
            memory: 256Mi
        limits:
            cpu: 500m
            memory: 512Mi

# ============================================================================
# NODE AGENTS CONFIGURATION
# ============================================================================
# Node agents run as a DaemonSet (one pod per node) and collect:
# - Container metrics and logs
# - Host metrics (CPU, memory, disk, network)
# - APM traces
# - Process information
agents:
    enabled: true
    
    # Rolling update strategy ensures gradual rollout across nodes
    updateStrategy:
        type: RollingUpdate
        rollingUpdate:
            maxUnavailable: 33%  # Update one node at a time for safety
    
    # Resource limits for each agent container to prevent resource exhaustion
    containers:
        # Main agent container - collects metrics, logs, and orchestrates other agents
        agent:
            resources:
                requests:
                    cpu: 200m
                    memory: 256Mi
                limits:
                    cpu: 500m
                    memory: 512Mi
        
        # Trace agent - handles APM trace ingestion and processing
        traceAgent:
            resources:
                requests:
                    cpu: 100m
                    memory: 128Mi
                limits:
                    cpu: 300m
                    memory: 256Mi
        
        # Process agent - collects process-level information
        processAgent:
            resources:
                requests:
                    cpu: 100m
                    memory: 128Mi
                limits:
                    cpu: 200m
                    memory: 256Mi

# ============================================================================
# DATADOG CONFIGURATION
# ============================================================================
datadog:
    # ========================================================================
    # TAGGING STRATEGY
    # ========================================================================
    # Tags applied to all metrics, traces, and logs for filtering/grouping
    tags:
        - env:development
    
    # ========================================================================
    # CLUSTER IDENTIFICATION
    # ========================================================================
    clusterName: development
    site: datadoghq.eu  # EU Datadog region
    apiKeyExistingSecret: datadog-api-key  # Kubernetes secret containing API key
    appKeyExistingSecret: datadog-app-key  # Kubernetes secret containing App key
    
    # ========================================================================
    # KUBERNETES STATE METRICS
    # ========================================================================
    # Collects cluster-level metrics (deployments, pods, nodes, etc.)
    kubeStateMetricsEnabled: true
    
    # ========================================================================
    # CLUSTER CHECKS
    # ========================================================================
    # Distributed checks across cluster agent pods for load balancing
    clusterChecks:
        enabled: true
    
    # ========================================================================
    # NETWORK PERFORMANCE MONITORING
    # ========================================================================
    # Network monitoring is disabled because enabling it causes issues in the
    # kubernetes cluster. https://github.com/DataDog/datadog-agent/issues/41350
    networkMonitoring:
        enabled: false
    
    # ========================================================================
    # LOG COLLECTION
    # ========================================================================
    logs:
        enabled: true
        containerCollectAll: true  # Collect logs from all containers
        containerCollectUsingFiles: true  # More reliable than Docker API
        
        # Scrub sensitive data from logs (passwords, tokens, etc.)
        containerScrubbing:
            enabled: true
        
        # Exclude noisy logs to reduce volume and costs
        containerExclude: "name:datadog-agent"  # Don't collect Datadog's own logs
        containerExcludeLogs: "kube_namespace:kube-system name:kube-.*"  # Reduce k8s system noise
        
        # Automatically detect multi-line logs (stack traces, JSON, etc.)
        autoMultiLineDetection: true
    
    # ========================================================================
    # APPLICATION PERFORMANCE MONITORING (APM)
    # ========================================================================
    apm:
        # Obfuscate sensitive data in traces (SQL queries, URLs, etc.)
        obfuscation:
            enabled: true
        
        # Automatic instrumentation injects APM libraries into pods at runtime
        instrumentation:
            enabled: true
            
            # Target specific namespaces for instrumentation
            targets:
                # Application services in development namespaces
                - name: 'services'
                  namespaceSelector:
                      matchNames:
                          - 'development-release'
                          - 'development-master'
                  ddTraceVersions:
                      python: '3'  # Use latest python tracer v3.x
                      js: '5'      # Use latest Node.js tracer v5.x
                
                # Instrument ingress controllers for request tracing
                - name: 'ingress'
                  namespaceSelector:
                      matchNames:
                          - 'ingress-nginx'
                          - 'alb-ingress-nginx'
    
    # ========================================================================
    # PROCESS MONITORING
    # ========================================================================
    # Collect process-level metrics and live process data
    processAgent:
        enabled: true
        processCollection: true  # Enable live process collection
    
    # ========================================================================
    # ORCHESTRATOR EXPLORER
    # ========================================================================
    # Provides visibility into Kubernetes resources (pods, deployments, etc.)
    orchestratorExplorer:
        enabled: true
    
    # ========================================================================
    # SECURITY MONITORING
    # ========================================================================
    securityAgent:
        # Compliance monitoring (CIS benchmarks, security best practices)
        compliance:
            enabled: true
            checkInterval: 30m  # Run compliance checks every 30 minutes
        
        # Runtime security monitoring (detects threats at runtime)
        runtime:
            enabled: true
            syscallMonitor:
                enabled: true  # Monitor system calls for security threats
    
    # ========================================================================
    # PROMETHEUS METRICS SCRAPING
    # ========================================================================
    # Automatically discover and scrape Prometheus metrics from pods
    prometheusScrape:
        enabled: true
        serviceEndpoints: true  # Scrape service endpoints
    
    # ========================================================================
    # HELM RELEASE MONITORING
    # ========================================================================
    # Monitor Helm releases and collect deployment events
    helmCheck:
        enabled: true
        collectEvents: true  # Collect Helm release events
    
    # ========================================================================
    # CONTAINER LIFECYCLE MONITORING
    # ========================================================================
    # Track container lifecycle events (start, stop, restart)
    containerLifecycle:
        enabled: true

The datadog.networkMonitoring.enabled is set to false. Hoping that this will mitigate the issue, we are going to set datadog.securityAgent.runtime.network.enabled to false as well.
If you have any other suggestions of networking features we could disable in the meantime, it would be greatly appreciated.

Please let us know if there's anything else we could do to help you with the investigation, we would be more than happy to help - we're really keen on adopting Datadog long-term 😄

[paulcacheux]

Hello @iuliancmarcu ,
My apologies for the delay since your last message. Did you see the issue again after setting datadog.securityAgent.runtime.network.enabled to false ?

I investigated a bit your issue, and I have a few questions. Is your EKS cluster configured in IPv6 only mode ?
In addition would you be able to run

ip addr show

in one of your pods (if ip is intalled), otherwise just:

cat /proc/net/dev

should be enough.

To fix your issue, we believe setting event_monitoring_config.network.lazy_interface_prefixes to v4if0 should help mitigate your issue. This can be done in your help chart using:

agents:
  containers:
    systemProbe:
      env:
      - name: DD_EVENT_MONITORING_CONFIG_NETWORK_LAZY_INTERFACE_PREFIXES
        value: "v4if0"

Let me know if this helps !
Thanks a lot for your report, and apologies again for the issue.