node-exporter in CrashLoopBackOff state #790

Louispettitt · 2020-11-16T17:37:33Z

Louispettitt
Nov 16, 2020

What happened?

Hello, I am new to K8s and have been tasked with setting up the monitoring stack for a cluster which is hosted on AWS EKS.

I followed steps in the Quick start and all of the pods ran apart from node-exporter, which gives out the following message when I run kubectl get pods -n monitoring:

CrashLoopBackOff

I cannot for the life of me work out how to right this issue. I have gone through various issues to see if I can find something similar to help me with the fix but to no avail. I am probably missing something basic here because I am a K8s novice, so any advice that you can give me or if you can point me in the right direction for a fix would be much appreciated!

Im pretty sure we meet the pre-requisite requirements with the authentication and authorisation webhook and that the correct Kube-prom stack is being used (release-0.6) for our cluster.

Did you expect to see some different?

All pods to be working.

How to reproduce it (as minimally and precisely as possible):

Quick start

kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/

Environment

Prometheus Operator version:

quay.io/prometheus-operator/prometheus-operator:v0.42.1

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Kubernetes cluster kind:

I am not sure how this was created but I can find out if need be.

I know we are using AWS EKS.

Manifests:

node-exporter-daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/name: node-exporter
    app.kubernetes.io/version: v0.18.1
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: node-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: node-exporter
        app.kubernetes.io/version: v0.18.1
    spec:
      containers:
      - args:
        - --web.listen-address=127.0.0.1:9100
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        - --no-collector.wifi
        - --no-collector.hwmon
        - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
        image: quay.io/prometheus/node-exporter:v0.18.1
        name: node-exporter
        resources:
          limits:
            cpu: 250m
            memory: 180Mi
          requests:
            cpu: 102m
            memory: 180Mi
        volumeMounts:
        - mountPath: /host/proc
          name: proc
          readOnly: false
        - mountPath: /host/sys
          name: sys
          readOnly: false
        - mountPath: /host/root
          mountPropagation: HostToContainer
          name: root
          readOnly: true
      - args:
        - --logtostderr
        - --secure-listen-address=[$(IP)]:9100
        - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - --upstream=http://127.0.0.1:9100/
        env:
        - name: IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        image: quay.io/coreos/kube-rbac-proxy:v0.4.1
        name: kube-rbac-proxy
        ports:
        - containerPort: 9100
          hostPort: 9100
          name: https
        resources:
          limits:
            cpu: 20m
            memory: 40Mi
          requests:
            cpu: 10m
            memory: 20Mi
      hostNetwork: true
      hostPID: true
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: node-exporter
      tolerations:
      - operator: Exists
      volumes:
      - hostPath:
          path: /proc
        name: proc
      - hostPath:
          path: /sys
        name: sys
      - hostPath:
          path: /
        name: root
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%

Prometheus Operator Logs:

kubectl logs node-exporter-gd5s6 -n monitoring kube-rbac-proxy
I1116 15:47:22.595166 17823 main.go:213] Generating self signed cert as no cert is provided
I1116 15:48:47.990746 17823 main.go:243] Starting TCP socket on [192.168.65.3]:9100
I1116 15:48:47.991641 17823 main.go:250] Listening securely on [192.168.65.3]:9100
2020/11/16 15:48:53 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused
2020/11/16 15:49:07 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused
2020/11/16 15:49:22 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused
2020/11/16 15:49:37 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused
2020/11/16 15:49:52 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused

kubectl logs node-exporter-gd5s6 -n monitoring node-exporter
failed to try resolving symlinks in path "/var/log/pods/monitoring_node-exporter-gd5s6_dcdde6b2-2f40-4fbb-b1a5-ff388057cbfb/node-exporter/8.log": lstat /var/log/pods/monitoring_node-exporter-gd5s6_dcdde6b2-2f40-4fbb-b1a5-ff388057cbfb/node-exporter/8.log: no such file or directory

kubectl describe pod node-exporter-gd5s6 -n monitoring

Name:         node-exporter-gd5s6
Namespace:    monitoring
Priority:     0
Node:         docker-desktop/192.168.65.3
Start Time:   Mon, 16 Nov 2020 15:47:01 +0000
Labels:       app.kubernetes.io/name=node-exporter
              app.kubernetes.io/version=v0.18.1
              controller-revision-hash=69cc88b55f
              pod-template-generation=1
Annotations:  <none>
Status:       Running
IP:           192.168.65.3
IPs:
  IP:           192.168.65.3
Controlled By:  DaemonSet/node-exporter
Containers:
  node-exporter:
    Container ID:  docker://e552750027a6c7d97457919857f85f41c04863ee76a309e4719f1d5c73752aee
    Image:         quay.io/prometheus/node-exporter:v0.18.1
    Image ID:      docker-pullable://quay.io/prometheus/node-exporter@sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee
    Port:          <none>
    Host Port:     <none>
    Args:
      --web.listen-address=127.0.0.1:9100
      --path.procfs=/host/proc
      --path.sysfs=/host/sys
      --path.rootfs=/host/root
      --no-collector.wifi
      --no-collector.hwmon
      --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      path / is mounted on / but it is not a shared or slave mount
      Exit Code:    128
      Started:      Mon, 16 Nov 2020 17:09:45 +0000
      Finished:     Mon, 16 Nov 2020 17:09:45 +0000
    Ready:          False
    Restart Count:  21
    Limits:
      cpu:     250m
      memory:  180Mi
    Requests:
      cpu:        102m
      memory:     180Mi
    Environment:  <none>
    Mounts:
      /host/proc from proc (rw)
      /host/root from root (ro)
      /host/sys from sys (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from node-exporter-token-l8d7w (ro)
  kube-rbac-proxy:
    Container ID:  docker://c79da379c9d53915a427ae9def06900293957f5c2dcd1614a0eaf17319a6995c
    Image:         quay.io/coreos/kube-rbac-proxy:v0.4.1
    Image ID:      docker-pullable://quay.io/coreos/kube-rbac-proxy@sha256:9d07c391aeb1a9d02eb4343c113ed01825227c70c32b3cae861711f90191b0fd
    Port:          9100/TCP
    Host Port:     9100/TCP
    Args:
      --logtostderr
      --secure-listen-address=[$(IP)]:9100
      --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
      --upstream=http://127.0.0.1:9100/
    State:          Running
      Started:      Mon, 16 Nov 2020 15:47:14 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     20m
      memory:  40Mi
    Requests:
      cpu:     10m
      memory:  20Mi
    Environment:
      IP:   (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from node-exporter-token-l8d7w (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  proc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  
  sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  
  root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  
  node-exporter-token-l8d7w:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  node-exporter-token-l8d7w
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason   Age                   From                     Message
  ----     ------   ----                  ----                     -------
  Warning  BackOff  116s (x383 over 86m)  kubelet, docker-desktop  Back-off restarting failed container

Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?:

I am using Mac.

docker version
Client: Docker Engine - Community
Cloud integration: 1.0.2
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 16:58:31 2020
OS/Arch: darwin/amd64
Experimental: false

Also, some screen shots for reference from Docker and Prometheus UI:

Ignore that the pod name doesn't match the pods mentioned above as I re-deployed the stack after - error message still the same.

Answered by yamenkaba

Sep 3, 2023

You can use this as a workaround,

Find Daemon Set name of the node-exporter
kubectl get ds -n <namespace>
Patch it with the following command
kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

View full answer

paulfantom · 2020-11-17T09:17:14Z

paulfantom
Nov 17, 2020
Maintainer

I believe you run into kubernetes/kubernetes#61058. For Linux, remounting / with mount --make-rshared / should solve it. I have no idea about darwin.

0 replies

Louispettitt · 2020-11-17T11:54:10Z

Louispettitt
Nov 17, 2020
Author

So I'd need to find and use the equivalent of mount --make-rshared / for darwin?

Screenshot of attempt for reference:

0 replies

paulfantom · 2020-11-20T09:31:08Z

paulfantom
Nov 20, 2020
Maintainer

Either this or deploy it in VM with Linux. For example in minikube.

0 replies

4n3w · 2020-11-23T19:22:19Z

4n3w
Nov 23, 2020

Curious: What side effects would changing the mountPropagation to None have on the NodeExporter? I have some objections from administrators that don't want to allow the mountPropagation to support `HostToContainer, ex:

   - mountPath: /host/root
      mountPropagation: None
      name: root
      readOnly: true

4 replies

paulfantom Nov 24, 2020
Maintainer

Without propagation for /host/root you won't get metrics for filesystem collector. Without propagation for /host/sys and /host/proc you won't get other metrics for node_exporter. In essence, you won't get all metrics for the node but only some metrics from a container.

gliptak Jan 27, 2021

I see similar issue using Windows Docker Desktop (built-in Kubernetes). The message was different:

      Message:      path /sys is mounted on /sys but it is not a shared or slave mount
      Exit Code:    128

I used script from https://github.com/luksa/kubectl-plugins/ to run mount --make-rshared /sys to no avail.

#601

paulfantom Jan 28, 2021
Maintainer

@gliptak I don't have much knowledge in Windows Docker Desktop, but it seems like you are hitting kubernetes/kubernetes#61058

gliptak Jan 28, 2021

@paulfantom thank you for the pointer. I think that is where I found the mount command referenced

Jamil100 · 2021-06-27T08:41:45Z

Jamil100
Jun 27, 2021

Was someone able to solve this problem?

2 replies

AlaaHamoudah Jul 4, 2021

There is a workaround to remove the mountPropagation to use the default one through creating a patch for promethous exporter damonset, here is the solution

paulfantom Jul 5, 2021
Maintainer

Bear in mind that by disabling mount propagation node-exporter won't be able to collect all data needed for grafana dashboards and for alerts.

TahoorYaseen · 2023-01-02T10:46:43Z

TahoorYaseen
Jan 2, 2023

@Louispettitt I ran into similar issue with docker desktop kubernetes cluster v1.22 and kube-prometheus release 0.9 and did you find the solution for it ?

0 replies

yamenkaba · 2023-09-03T11:54:47Z

yamenkaba
Sep 3, 2023

You can use this as a workaround,

Find Daemon Set name of the node-exporter
kubectl get ds -n <namespace>
Patch it with the following command
kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

1 reply

Sounak3007 May 22, 2024

thanks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

node-exporter in CrashLoopBackOff state #790

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

node-exporter in CrashLoopBackOff state #790

Replies: 7 comments · 7 replies

paulfantom Nov 17, 2020 Maintainer

Louispettitt Nov 17, 2020 Author

paulfantom Nov 20, 2020 Maintainer

paulfantom Nov 24, 2020 Maintainer

paulfantom Jan 28, 2021 Maintainer

paulfantom Jul 5, 2021 Maintainer

Replies: 7 comments 7 replies

paulfantom
Nov 17, 2020
Maintainer

Louispettitt
Nov 17, 2020
Author

paulfantom
Nov 20, 2020
Maintainer

paulfantom Nov 24, 2020
Maintainer

paulfantom Jan 28, 2021
Maintainer

paulfantom Jul 5, 2021
Maintainer