linkerd-viz helm: add support for metric_relabel_configs (#12248)

It is currently not possible to easily drop metrics with the Prometheus configuration that is included with the linkerd-viz helm chart. To enable this we extend the Prometheus configuration template to add a metric_relabel_config section. This configuration is usually used when dropping metrics and thus allows users to customize their recorded metrics to avoid high cardinality. E.g. there is a blog post that describes the problem in depth: https://itnext.io/optimizing-linkerd-metrics-in-prometheus-de607ec10f6b With this change we are able to deploy the helm chart without the need to do custom modifications to the templates. Fixes #12194 Signed-off-by: Heiko Voigt <[email protected]>
linkerd · Apr 11, 2024 · f283fc3 · f283fc3
1 parent 340e425
commit f283fc3
Show file tree

Hide file tree

Showing 3 changed files with 17 additions and 0 deletions.
diff --git a/viz/charts/linkerd-viz/README.md b/viz/charts/linkerd-viz/README.md
@@ -141,6 +141,7 @@ Kubernetes: `>=1.22.0-0`
 | prometheus.image.tag | string | `"v2.48.1"` | Docker image tag for the prometheus instance |
 | prometheus.logFormat | string | defaultLogLevel | log format (plain, json) of the prometheus instance |
 | prometheus.logLevel | string | defaultLogLevel | log level of the prometheus instance |
+| prometheus.metricRelabelConfigs | string | `nil` | A metricRelabelConfigs section allows to drop high cardinality metrics. *NOTE:* Please use with caution. Some metrics are needed for linkerd-viz to function properly. |
 | prometheus.nodeSelector | object | `{"kubernetes.io/os":"linux"}` | NodeSelector section, See the [K8S documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) for more information |
 | prometheus.podAnnotations | object | `{}` | annotations for the prometheus pod |
 | prometheus.proxy | string | `nil` |  |

diff --git a/viz/charts/linkerd-viz/templates/prometheus.yaml b/viz/charts/linkerd-viz/templates/prometheus.yaml
@@ -137,6 +137,10 @@ data:
       # Copy tmp labels into real labels
       - action: labelmap
         regex: __tmp_pod_label_(.+)
+      {{- if .Values.prometheus.metricRelabelConfigs }}
+      metric_relabel_configs:
+      {{- toYaml .Values.prometheus.metricRelabelConfigs | trim | nindent 6 }}
+      {{- end}}
 
     {{- if .Values.prometheus.scrapeConfigs }}
     {{- toYaml .Values.prometheus.scrapeConfigs | trim | nindent 4 }}

diff --git a/viz/charts/linkerd-viz/values.yaml b/viz/charts/linkerd-viz/values.yaml
@@ -499,6 +499,18 @@ prometheus:
   #   - action: labelmap
   #     regex: __meta_kubernetes_node_label_(.+)
 
+  # -- A metricRelabelConfigs section allows to drop high cardinality metrics.
+  # *NOTE:* Please use with caution. Some metrics are needed for linkerd-viz to
+  # function properly.
+  metricRelabelConfigs:
+  # This allows us to fine tune prometheus cardinality by dropping certain
+  # metrics as suggested here: https://itnext.io/optimizing-linkerd-metrics-in-prometheus-de607ec10f6b
+  #
+  # Ex:
+  # - action: keep
+  #   source_labels: [le]
+  #   regex: "(?i)(|10|50|100|500|1000|10000|30000|\\+Inf)"
+
   # -- A sidecarContainers section specifies a list of secondary containers to run
   # in the prometheus pod e.g. to export data to non-prometheus systems
   sidecarContainers: