unable to fetch metrics from external metrics API: Internal error occurred: DatadogMetric is invalid, err: Invalid metric (from backend)

[cforce]

{"level":"info","ts":1624011129.786673,"logger":"controllers.WatermarkPodAutoscaler","msg":"Failed to compute desired number of replicas based on listed metrics.","watermarkpodautoscaler":"dev/myapp","reference":"Deployment/dev/myapp","error":"failed to get external metric kubernetes.cpu.usage: unable to get external metric dev/kubernetes.cpu.usage/&LabelSelector{MatchLabels:map[string]string{app: myapp,release: myapp,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: Internal error occurred: DatadogMetric is invalid, err: Invalid metric (from backend), query: avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)"}
{"level":"info","ts":1624011144.793971,"logger":"controllers.WatermarkPodAutoscaler","msg":"Target deploy","watermarkpodautoscaler":"dev/myapp","replicas":2}
{"level":"info","ts":1624011144.7941537,"logger":"controllers.WatermarkPodAutoscaler","msg":"getReadyPodsCount","watermarkpodautoscaler":"dev/myapp","full podList length":2,"toleratedAsReadyPodCount":2,"incorrectly targeted pods":0}
{"level":"info","ts":1624011144.8295028,"logger":"controllers.WatermarkPodAutoscaler","msg":"Failed to compute desired number of replicas based on listed metrics.","watermarkpodautoscaler":"dev/myapp","reference":"Deployment/dev/myapp","error":"failed to get external metric kubernetes.cpu.usage: unable to get external metric dev/kubernetes.cpu.usage/&LabelSelector{MatchLabels:map[string]string{app: myapp,release: myapp,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: Internal error occurred: DatadogMetric is invalid, err: Invalid metric (from backend), query: avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)"}
{"level":"info","ts":1624011159.8374639,"logger":"controllers.WatermarkPodAutoscaler","msg":"Target deploy","watermarkpodautoscaler":"dev/myapp","replicas":2}

Ask for metric (without tags)

% kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/dev/kubernetes.cpu.usage | jq ."
Error from server (InternalError): Internal error occurred: DatadogMetric not found for metric name: kubernetes.cpu.usage | jq ., datadogmetricid: datadog/dcaautogen-646a73ad876299907eb8035a2fa8e2b60ac832

Impersonate and ask for any metric

% kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/dev/metric --as system:serviceaccount:dev:watermarkpodautoscaler"
Error from server (InternalError): Internal error occurred: DatadogMetric not found for metric name: metric --as system:serviceaccount:dev:watermarkpodautoscaler, datadogmetricid: datadog/dcaautogen-98eae079d0f4a80135d6f4b6f9762cea878b97

What did i wrong?
How can i general find out which metrics and which tags for it are available?

[vboulineau]

Hello @cforce,

It normally means that no data is available for avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)", you can put this query in a Notebook to visualize the data used for autoscaling.

If you'd like to see what the API returns, you can use a small Python script using our Datadog client:

import time
import json

from datadog import initialize, api

options = {
    'api_key': '<api_key>',
    'app_key': '<app_key>'
}

initialize(**options)

print(json.dumps(api.Metric.query(start=int(time.time()) - 10, end=int(time.time()), query='avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)'), sort_keys=True, indent=4))

[cforce]

I used this modified script:

import time
import json
import requests
import sys
import json
import argparse
from datadog import initialize, api

print('Executing script file is:', str(sys.argv[0]))
print('The arguments are:', str(sys.argv))
parser = argparse.ArgumentParser()
parser.add_argument("--apikey", help="Provide the DataDog ApiKey.", required=True)
parser.add_argument("--appkey", help="Provide the DataDog AppKey.", required=True)
parser.add_argument("--query", help="Provide the metric query to check, e.g 'avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)'", required=True)

args = parser.parse_args()
apiKey = args.apikey
appKey = args.appkey
query = args.query

options = {
    'api_key': apiKey,
    'app_key': appKey,
    'application_key': appKey
}

initialize(**options)

print(json.dumps(api.Metric.query(start=int(time.time()) - 10, end=int(time.time()), query=query), sort_keys=True, indent=4))

Response:

{
    "from_date": 1624114030000,
    "group_by": [],
    "message": "",
    "query": "avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)",
    "res_type": "time_series",
    "resp_version": 1,
    "series": [],
    "status": "ok",
    "times": [],
    "to_date": 1624114040000,
    "values": []
}

Seems there are no values? So the tags does not fit - how to try out without and howto find out which are available?
I tried with

• avg:kubernetes.cpu.usage{service:myapp}.rollup(30) - also values[]
• avg:kubernetes.cpu.usage{}.rollup(30) - failure
• avg:kubernetes.cpu.usage.rollup(30) - failure
• avg:kubernetes.cpu.usage - failure

[vboulineau]

That's the expected behaviour indeed. About avg:kubernetes.cpu.usage{}.rollup(30), querying without any tag is avg:kubernetes.cpu.usage{*}.rollup(30).

About checking available tags, you have 2 tools in Datadog to explore metrics/tags:

• Metrics explorer
• Notebooks

Now if you want to have all possible tags for a metric, you can check the Metrics summary (for instance for kubernetes.cpu.usage.total https://app.datadoghq.com/metric/summary?filter=kubernetes.cpu.&metric=kubernetes.cpu.usage.total)