Bug: metricUnavailableValue parameter on datadog-scaler is non-functional in KEDA 2.18.1

[matchan26]

Report

In KEDA 2.18.1, the metricUnavailableValue parameter for the Datadog scaler is completely non-functional. Regardless of what value is set (including non-zero values), the scaler always throws an error when Datadog returns no metrics, instead of using the configured fallback value.

This is a regression introduced in KEDA 2.18.0/2.18.1 that breaks backward compatibility with KEDA 2.17.2.

Environment:

• KEDA Version: 2.18.1
• Kubernetes Version: 1.31
• Platform: Amazon EKS
• Scaler: Datadog (REST API mode, not cluster agent proxy)

Expected Behavior

When metricUnavailableValue is set to any valid value (e.g., "0", "1", "-1", etc.), and Datadog returns no metrics for the given time window:

1. KEDA should use the configured metricUnavailableValue as the metric value
2. No error should be thrown
3. HPA should continue to operate normally with the fallback value

This was the working behavior in KEDA 2.17.2, where the parameter being set (regardless of its value) would enable the fallback mechanism.

Actual Behavior

When metricUnavailableValue is set to ANY value (including non-zero values), and Datadog returns no metrics:

1. KEDA throws error: "no Datadog metrics returned for the given time window"
2. ScaledObject status becomes Degraded
3. HPA shows FailedGetExternalMetric condition
4. The configured metricUnavailableValue is completely ignored

The parameter has no effect at all in KEDA 2.18.1, regardless of its value:

metricUnavailableValue	FillValue	UseFiller	Result when series is empty
Not set	0 (default)	false	❌ Error thrown
"0"	0	false	❌ Error thrown
"1"	1	false	❌ Error thrown (BUG)
"-1"	-1	false	❌ Error thrown (BUG)
"100"	100	false	❌ Error thrown (BUG)

Steps to Reproduce the Problem

Step 1: Deploy ScaledObject with metricUnavailableValue

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: datadog-test
  namespace: test
spec:
  scaleTargetRef:
    name: test-deployment
  minReplicaCount: 1
  maxReplicaCount: 5
  triggers:
  - type: datadog
    metadata:
      query: "max:sidekiq.test.queue_latency{*}.rollup(60)"
      queryValue: "120"
      age: "60"
      metricUnavailableValue: "1"  # Non-zero value
      activationQueryValue: "-1"
    authenticationRef:
      name: keda-datadog-auth
      kind: ClusterTriggerAuthentication

Step 2: Wait for Datadog API to return empty series

This occurs when:

• The time window (age: "60") is too narrow for the metric's rollup interval
• There is a timing mismatch between metric ingestion and query execution
• Metrics are temporarily unavailable

Step 3: Observe the error

The ScaledObject will fail with errors, and HPA will be unable to get metrics.

Step 4: Verify with Direct API Call

# Datadog API returns empty series
curl -G "https://api.datadoghq.com/api/v1/query" \
  --data-urlencode "from=<timestamp>" \
  --data-urlencode "to=<timestamp>" \
  --data-urlencode "query=max:sidekiq.test.queue_latency{*}.rollup(60)" \
  -H "DD-API-KEY: <key>" \
  -H "DD-APPLICATION-KEY: <app-key>"

# Response:
{
  "status": "ok",
  "series": [],  # Empty - no data points
  ...
}

Logs from KEDA operator

{
	"content": {
		"timestamp": "2025-11-06T09:51:18.519Z",
		"tags": [
		],
		"host": "i-08401f2e76df56cc6",
		"service": "keda",
		"message": "error getting metrics from Datadog",
		"attributes": {
			"stacktrace": "github.com/kedacore/keda/v2/pkg/scalers.(*datadogScaler).GetMetricsAndActivity\n\t/workspace/pkg/scalers/datadog_scaler.go:496\ngithub.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsAndActivityForScaler\n\t/workspace/pkg/scaling/cache/scalers_cache.go:164\ngithub.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalerState\n\t/workspace/pkg/scaling/scale_handler.go:793\ngithub.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState.func1\n\t/workspace/pkg/scaling/scale_handler.go:662",
			"level": "error",
			"logger": "datadog_scaler",
			"name": "batch-scaledobject",
			"type": "ScaledObject",
			"error": "no Datadog metrics returned for the given time window",
			"ts": "2025-11-06T09:51:18Z"
		}
	}
}

KEDA Version

2.18.1

Kubernetes Version

1.31

Platform

Amazon Web Services

Scaler Details

datadog

Anything else?

Root Cause Analysis

The bug is in the Validate() method introduced in KEDA 2.18.x (PR #6721):

File: pkg/scalers/datadog_scaler.go
Line: 503

Buggy Code (KEDA 2.18.1)

// datadog_scaler.go - Struct definition
type datadogMetadata struct {
    // ...
    FillValue  float64 `keda:"name=metricUnavailableValue, order=triggerMetadata, default=0"`
    UseFiller  bool    // No keda tag - not automatically set during parsing
    // ...
}

// datadog_scaler.go:503 - Validate() method
func (s *datadogMetadata) Validate() error {
    // ... other validation ...
    if s.FillValue == 0 {
        s.UseFiller = false
    }
    // BUG: Missing else clause to set UseFiller = true for non-zero FillValue
    return nil
}

Error Triggering Code

// datadog_scaler.go:354-358
if len(series) == 0 {
    if !s.metadata.UseFiller {  // Always true because UseFiller = false
        return 0, fmt.Errorf("no Datadog metrics returned for the given time window")
    }
    return s.metadata.FillValue, nil  // Never reached
}

The Bug Explained

In KEDA 2.18.1:

1. During parsing: FillValue is populated from metricUnavailableValue parameter
2. During validation:
   • If FillValue == 0: UseFiller is explicitly set to false
   • If FillValue != 0: UseFiller remains at its default value (false) because there's no else clause
3. During metric retrieval:
   • When Datadog returns empty series, the code checks !UseFiller
   • Since UseFiller is always false, the error is always thrown
   • The FillValue is never used

Working Code (KEDA 2.17.2)

// In KEDA 2.17.2, during parsing:
if val, ok := config.TriggerMetadata["metricUnavailableValue"]; ok {
    fillValue, err := strconv.ParseFloat(val, 64)
    if err != nil {
        return nil, fmt.Errorf("metricUnavailableValue parsing error %s", err.Error())
    }
    meta.fillValue = fillValue
    meta.useFiller = true  // Unconditionally set to true when parameter exists
}

Proposed Fix

The fix is straightforward - add an else clause to properly set UseFiller:

// datadog_scaler.go:503
func (s *datadogMetadata) Validate() error {
    // ... other validation ...
    if s.FillValue == 0 {
        s.UseFiller = false
    } else {
        s.UseFiller = true  // FIX: Set to true for non-zero values
    }
    return nil
}

Or more concisely:

func (s *datadogMetadata) Validate() error {
    // ... other validation ...
    s.UseFiller = (s.FillValue != 0)
    return nil
}

Impact Assessment

• Severity: High - Complete loss of fallback functionality
• Scope: All Datadog scaler users relying on metricUnavailableValue
• Backward Compatibility: Breaking change from 2.17.2
• Version Comparison:
  • KEDA 2.17.2: ✅ metricUnavailableValue works correctly
  • KEDA 2.18.0: ❌ Bug introduced (PR Add Error for unmatched optional property. #6721)
  • KEDA 2.18.1: ❌ Bug still present

[rickbrouwer]

Thanks for the report! Since you're proposing a fix, would you consider offering it in a PR?

[matchan26]

Thanks for the quick reply!

The proposed fix in the original issue comment lacked consideration for when a user explicitly sets it to 0 (thinking about it, our case falls under this). So, I'm going to change the approach slightly and create a PR. Please wait a little longer.