Docker allocations stats (CPU+RAM) not collected by Nomad client

[equals215]

Versions

Nomad

Nomad v1.10.2
BuildDate 2025-06-09T22:00:49Z
Revision df4c764f6703e99e4df6f4d6c46c916d97ef8f84

Consul

Consul v1.21.2
Revision 136b9cb8
Build Date 2025-06-18T08:16:39Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Docker

Client: Docker Engine - Community
 Version:           28.3.0
 API version:       1.51
 Go version:        go1.24.4
 Git commit:        38b7060
 Built:             Tue Jun 24 15:44:25 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.3.0
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.4
  Git commit:       265f709
  Built:            Tue Jun 24 15:44:25 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Operating system and Environment details

qemu virtualized Debian 12 kept fully up to date
AMD EPYCs CPUs and ECC ram
--->Nomad clients were initially started running as nomad user, then switched to root at the same time as upgrading from v1.10.1 to v.10.2<--- might be a reason

Issue

Nomad WebUI doesn't show stats for allocations
API calls from the WebUI return :

{
    "ResourceUsage": {
        "CpuStats": {
            "Measured": null,
            "Percent": 0,
            "SystemMode": 0,
            "ThrottledPeriods": 0,
            "ThrottledTime": 0,
            "TotalTicks": 0,
            "UserMode": 0
        },
        "DeviceStats": [],
        "MemoryStats": {
            "Cache": 0,
            "KernelMaxUsage": 0,
            "KernelUsage": 0,
            "MappedFile": 0,
            "MaxUsage": 0,
            "Measured": null,
            "RSS": 0,
            "Swap": 0,
            "Usage": 0
        }
    },
    "Tasks": {},
    "Timestamp": 0
}

nomad alloc status shows only the provisioned resources like :

CPU       Memory   Disk     Addresses
1000 MHz  2.0 GiB  300 MiB 

but docker stats on the client node works well and gives all the info :

CONTAINER ID   NAME                                          CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS 
d8478e58c017   xxxx   0.02%     692.6MiB / 2GiB     33.82%    20kB / 14.5kB   0B / 0B    141

Reproduction

No idea how to reproduce this

Logs

Logs are not relevant and even with log_level="TRACE" nothing abnormal shows up

[equals215]

On a cloned client VM with the following versions :

Nomad v1.10.1
BuildDate 2025-05-13T07:40:43Z
Revision 3431f13e8036b4716aac0e3b8c5854ddca212e5c

Consul v1.21.1
Revision 48a6c3e4
Build Date 2025-05-23T04:53:32Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Client: Docker Engine - Community
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:26 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       45873be
  Built:            Fri May 30 12:07:26 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Everything works as expected, will try to downgrade a node service by service to find which service is responsible for the metrics not showing

[equals215]

On the cloned client VM :
Docker was bumped from 28.2.2 to 28.3.0 : no changes CPU&RAM stats collection still works
Consul was bumped from v1.21.1 to v1.21.2 : no changes CPU&RAM stats collection still works
Nomad was bumped from v1.10.1 to v1.10.2 : CPU&RAM stats collection is not working anymore

[equals215]

On another client VM downgrading Nomad from v1.10.2 to v1.10.1 restored stats collection without causing other problems

[tgross]

@equals215 I suspect you've got telemetry.publish_allocation_metrics=false (or unset) in your client configurations. In #25870 which shipped in 1.10.2, we fixed a bug where metrics were being collected when they shouldn't be. I didn't think they were being published too, but that seems to be the case with what you're reporting here.

If telemetry.publish_allocation_metrics=true on the clients fixes things for you, then that means the bug we fixed was actually a bit wider than I thought. We should probably add a note about this to the upgrade guide because it was a fairly old bug.