Skip to content

node_exporter not reporting filesystem metrics for additional volume mounts in Docker #3194

@calixtoyg

Description

@calixtoyg

Host operating system: output of uname -a

Linux calixtoubuntuserver 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov  4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.8.2 (branch: HEAD, revision: f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)
  build user:       root@03d440803209
  build date:       20240714-11:53:45
  go version:       go1.22.5
  platform:         linux/amd64
  tags:             unknown

node_exporter command line flags

--collector.filesystem
--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+|run/.+|/snap/.+)($|/)

node_exporter log output

ts=2024-11-30T11:24:24.674Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.2, branch=HEAD, revision=f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)"
ts=2024-11-30T11:24:24.674Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.5, platform=linux/amd64, user=root@03d440803209, date=20240714-11:53:45, tags=unknown)"
ts=2024-11-30T11:24:24.677Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+|run/.+|/snap/.+)($|/)
ts=2024-11-30T11:24:24.677Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-11-30T11:24:24.677Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2024-11-30T11:24:24.678Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=arp
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=bcache
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=bonding
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=btrfs
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=conntrack
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=cpu
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=cpufreq
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=diskstats
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=dmi
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=edac
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=entropy
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=fibrechannel
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=filefd
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=filesystem
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=hwmon
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=infiniband
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=ipvs
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=loadavg
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=mdadm
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=meminfo
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=netclass
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=netdev
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=netstat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=nfs
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=nfsd
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=nvme
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=os
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=powersupplyclass
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=pressure
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=rapl
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=schedstat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=selinux
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=sockstat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=softnet
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=stat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=tapestats
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=textfile
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=thermal_zone
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=time
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=timex
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=udp_queues
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=uname
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=vmstat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=watchdog
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=xfs
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=zfs
ts=2024-11-30T11:24:24.680Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
ts=2024-11-30T11:24:24.680Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100

Are you running node_exporter in Docker?

Yes

What did you do that produced an error?

  1. Mounted a volume /home/calixtoubuntu/Media:/media:ro in the node_exporter container
  2. The mount is visible inside the container in two places? When running docker exec node_exporter df -h | grep media -A 1:
                        292.8G    144.0G    135.8G  51% /rootfs/media
/dev/mapper/ubuntu--vg-ubuntu--lv
--
                        292.8G    144.0G    135.8G  51% /media
/dev/mapper/ubuntu--vg-ubuntu--lv

  1. However, when querying Prometheus for filesystem metrics, the mount is not reported

What did you expect to see?

The filesystem metrics for the mounted volume at /media showing up in Prometheus queries like:
node_filesystem_avail_bytes{device="/dev/mapper/ubuntu--vg-ubuntu--lv", mountpoint="/media"}

What did you see instead?

Only the root filesystem metrics are reported:

node_filesystem_avail_bytes{device="/dev/mapper/ubuntu--vg-ubuntu--lv", fstype="ext4", instance="host.docker.internal:9100", job="node_exporter", mountpoint="/"} [value]

No metrics are reported for the /media mountpoint despite it being correctly mounted in the container.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions