[bug] Broken panels on k8s-views-nodes

[reefland]

Describe the bug

I've read both issues in the README about broken panels on k8s-views-nodes dashboard.

• My nodes have fixed IP addresses so it's not 1st issues.
• The kube_node_info{node} matches node_uname_info{nodename} values, so doesn't seem like 2nd issue:

When reviewing the instance variable, the label filter for nodename has a regex filter which finds nothing. If I set it to be just $node -- all panels render fine.

This works great:

This default value has broken panels:

Suggestions?

Running K3s: v1.32.5+k3s1 on bare metal Ubuntu 25.04.

How to reproduce?

No response

Expected behavior

No response

Additional context

No response

[uhthomas]

This is because your metrics are missing the cluster label. Your fix works because =~"" actually matches everything, not because there is a valid value for $node and the regex is broken. You must add a cluster label to your metrics and it will work properly.

[reefland]

Following your advice, I added this to prometheus:

prometheusSpec:

    scrapeClasses:
      - default: true
        name: cluster-relabeling
        relabelings:
          - sourceLabels: [ __name__ ]
            regex: (.*)
            targetLabel: cluster
            replacement: k3s-prod
            action: replace

The cluster field is now populated, but many panels are still not working:

The instance variable still does not return any matches:

When I click [Run Query], button I get a popup:

[uhthomas]

Can you check that kube_node_info and node_uname_info definitely have the cluster label?

[reefland]

Can you check that kube_node_info and node_uname_info definitely have the cluster label?

[reefland]

And also to confirm we're using the same dashboard revision:

    kubernetes-nodes:
      # renovate: depName="Kubernetes / Views / Nodes"
      gnetId: 15759
      revision: 35
      datasource: Prometheus

[jasonpanosso]

I think I'm having a similar issue, however I believe mine is due to the escaping of the backslash in the regex. If I change the regex used in the instance variable to (?i:($node)(\\.[a-z0-9.]+)?)(add an extra backslash) my query works fine, otherwise I'm spammed with warnings/errors regarding 1:27: parse error: unknown escape sequence U+002E '.'

Can you check that kube_node_info and node_uname_info definitely have the cluster label?

What are the current values of the instance label on your node_uname_info metric? I was having issues with it not being aligned with the metrics being used in the panels, i.e. node_cpu_seconds_total.

edit: apologies for the noise, the dashboard works properly for me when using the raw url https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json, but I do have an issue regarding backslash escaping when using gnetID 15759 revision 35. Inspecting the JSON from the dashboard fetched from grafana instead of the main gh branch shows that the variable isn't properly escaped:

      {
        "current": {
          "text": "192.168.20.101",
          "value": "192.168.20.101"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "definition": "label_values(node_uname_info{nodename=~\"(?i:($node)(\\.[a-z0-9.]+)?)\"}, instance)",
        "hide": 2,
        "includeAll": false,
        "name": "instance",
        "options": [],
        "query": {
          "query": "label_values(node_uname_info{nodename=~\"(?i:($node)(\\.[a-z0-9.]+)?)\"}, instance)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 2,
        "regex": "",
        "sort": 1,
        "type": "query"
      }

dashboards:
  kubernetes:
    kubernetes-nodes:
      gnetId: 15759
      revision: 35
      datasource: Prometheus

[reefland]

I think I'm having a similar issue, however I believe mine is due to the escaping of the backslash in the regex. If I change the regex used in the instance variable to (?i:($node)(\\.[a-z0-9.]+)?)(add an extra backslash) my query works fine.

This is it, I also added an additional slash (?i:($node)(\\.[a-z0-9.]+)?) and the query works fine and all panels start working again.

[uhthomas]

That was fixed though?

https://github.com/dotdc/grafana-dashboards-kubernetes/releases/tag/v2.7.3

[jasonpanosso]

That was fixed though?

https://github.com/dotdc/grafana-dashboards-kubernetes/releases/tag/v2.7.3

it's fixed for the repo, but I think the uploaded version on grafana.com is out of sync, try this:

curl -s https://grafana.com/api/dashboards/15759/revisions/latest/download | jq -r '.templating.list[] | select(.name=="instance").definition'

[reefland]

dit: apologies for the noise, the dashboard works properly for me when using the raw url https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json, but I do have an issue regarding backslash escaping when using gnetID 15759 revision 35. Inspecting the JSON from the dashboard fetched from grafana instead of the main gh branch shows that the variable isn't properly escaped:

$ curl -s https://grafana.com/api/dashboards/15759/revisions/35/download | jq -r '.templating.list[] | select(.name=="instance").definition'

label_values(node_uname_info{nodename=~"(?i:($node)(\.[a-z0-9.]+)?)"}, instance)

So it looks like @dotdc just has to push the updated dashboard to Grafana?