[Bug]: intel_gpu_top polling once per MILLIsecond (versus intended once-per-second?)

[emily-pesce]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.
• I have asked the AI at https://docs.frigate.video about my issue.

Describe the problem you are having

I am running a new install. I've noticed after some time, the System Metrics > GPU Usage stat goes to 0% and sticks there. This is for a intel/qsv environment.

The cams still work and CPU doesnt spike so it appear things are still "working". But, Ive also been working on some system hard crashes (which led me to this oddity). I am not sure they are connected but alas...

I am running proxmox 9 with frigate in a debian (6.12.41+deb13-amd64) VM (and running in docker within that VM).

It figured it might just been the newness of trixie and passing through the GPU.

But, in debugging, I found that in services.py the call that gets this info (pasted below) was spitting out insane amounts of information when I tried it. LLMs were confused. Finally I looked at the command's help and notices that the -s flag, which the LLMs, and maybe frigate, expected to be in seconds is now in milliseconds. That explains the crazy polling output as frigate (and me copying frigate) were asking for data every 1ms.

intel_gpu_top_command = [
        "timeout",
        "0.5s",
        "intel_gpu_top",
        "-J",
        "-o",
        "-",
        "-s",
        "1",
    ]

root@d12ac4500abc:/opt/frigate# intel_gpu_top -h   
intel_gpu_top - Display a top-like summary of Intel GPU usage

Usage: intel_gpu_top [parameters]

	The following parameters are optional:

	[-h]            Show this help text.
	[-J]            Output JSON formatted data.
	[-l]            List plain text data.
	[-o <file|->]   Output to specified file or '-' for standard out.
	**[-s <ms>]       Refresh period in milliseconds (default 1000ms).**
	[-L]            List all cards.

Given the LLM assumption I wonder if this was a second flag in the past (as "-s" would imply) and recently got move to ms? And then I was wondering if polling every 1ms unintentionally would overload/crash something causing the metric to flatline.

Anyway, not sure what the intent was but thought I'd file this as an fyi because 1ms seems extremely frequent.

Steps to reproduce

1. frigate docker container starts, all is well
2. in some n hours the intel-qsv metric in the UI goes to 0 and sticks there.

Version

0.16.1-e664cb2

In which browser(s) are you experiencing the issue with?

Version 18.6 (20621.3.11.11.3)

Frigate config file

ffmpeg:
  hwaccel_args: preset-intel-qsv-h264

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    stop_grace_period: 30s # allow enough time to shut down the various services
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "512mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb 
      - /dev/dri/renderD128:/dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/emily/frigate/config:/config
      - /mnt/frigate:/media/frigate
      - type: tmpfs 
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
      - "1984:1984" #go2rtc

Relevant Frigate log output

n/a

Relevant go2rtc log output

n/a

Operating system

Home Assistant OS

Install method

Docker Compose

Network connection

Wired

Camera make and model

multiple

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Hardware: Beelink EQi12 Intel® Core 1220P

[NickM-27]

Wow, that would be quite an interesting breaking change to just throw in there, I'd expect they'd have some kind of sanity check as 1ms seems way too frequent to even be a valid number. Your suggestion about this change also leading to the GPU stats being broken could absolutely be the case.

I am not sure how to validate this longer-term, to make sure we are not hit by a problem if this behavior changes again. Seems like we should go ahead and try and change this for now though targeting 0.16.2

[emily-pesce]

yes, at the very least "-s" should not mean milliseconds!
sounds good for 0.16.2
I don't know how to suggest an actual code change or I'd do it myself! Sorry!

[hawkeye217]

No worries. We've put up a fix here: #19984

[hawkeye217]

So even after changing this to a 1000ms poll, I can confirm that intel_gpu_top still ends up reporting 0 usage after a while. It seemed fine for about ~8 hours and then at some point overnight it stopped. So the stats issue does still seem like an issue with the Intel driver somewhere.

[hawkeye217]

Thanks, great find!