[Support]: 16.0-rc1 100% CPU & memory usage

[ShyneTurtle]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

My frigate instance has been running for 4days straight now, and frigate has been consuming more and more memory over time (memory leak ?) and when it reached high 80% usage on my system, the CPU usage started peaking as well, before returning to normal for both memory and CPU.

Then over the next day it started increasing again, and when i tried to connect the whole system was unresponsive. I barely managed to SSH into the server and check the usage using docker stat, which reported >90% usage by frigate for both CPU and memory.

I then rebooted the system through SSH, but i can only see the logs following the restart.

Steps to reproduce

1. Start the frigate instance with a few cameras running, and wait.

Version

16.0-rc1-rk

In which browser(s) are you experiencing the issue with?

not relevant

Frigate config file

detect:
  enabled: true
  fps: 10

detectors: # required
  rknn: # required
    type: rknn # required
    # number of NPU cores to use
    # 0 means choose automatically
    # increase for better performance if you have a multicore NPU e.g. set to 3 on rk3588
    num_cores: 3

model: # required
  # name of model (will be automatically downloaded) or path to your own .rknn model file
  # possible values are:
  # - deci-fp16-yolonas_s
  # - deci-fp16-yolonas_m
  # - deci-fp16-yolonas_l
  # - /config/model_cache/your_custom_model.rknn
  path: deci-fp16-yolonas_m
  # width and height of detection frames
  width: 320
  height: 320
  # pixel format of detection frame
  # default value is rgb but yolo models usually use bgr format
  input_pixel_format: bgr # required
  # shape of detection frame
  input_tensor: nhwc
  # needs to be adjusted to model, see below
  labelmap_path: /labelmap/coco-80.txt # required
objects:
  track:
    - person
  filters:
    person:
      # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
      threshold: 0.7

logger:
  default: critical

mqtt:
  host: mqtt
  user: frigate
  password: [REDACTED]

tls:
  enabled: false

record:
  enabled: true
  retain:
    days: 15
    mode: all
  alerts:
    retain:
      days: 30
  detections:
    retain:
      days: 30
ffmpeg:
  hwaccel_args: preset-rk-h264
  input_args: preset-rtsp-generic
  output_args:
    record: preset-record-generic-audio-aac

cameras:
  cam240:
    ffmpeg:
      inputs:
        - path: rtsp://[REDACTED]@192.168.2.240/av_stream/ch0
          roles:
            - detect
  cam241:
    ffmpeg:
      inputs:
        - path: rtsp://[REDACTED]@192.168.2.241/av_stream/ch0
          roles:
            - detect
    motion:
      mask:
        - 0.237,0.046,0.002,0.046,0,0,0.239,0.002
        - 0.675,0.728,0.622,0.364,0.776,0.306,0.812,0.694
        - 0.045,0.66,0.169,0.243,0.078,0.001,0,0.002,0,0.602
  cam242:
    ffmpeg:
      inputs:
        - path: rtsp://[REDACTED]@192.168.2.242/av_stream/ch0
          roles:
            - detect
    motion:
      mask:
        - 0.238,0.043,0.002,0.043,0.001,0.005,0.237,0.002
        - 0.472,0.462,0.892,0.437,0.984,0.242,0.988,0.008,0.351,0
    zones: {}
version: 0.16-0
semantic_search:
  enabled: true
  reindex: false
  model_size: small

docker-compose file or Docker CLI command

services:
  frigate:
    image: ghcr.io/blakeblackshear/frigate:0.16.0-rc1-rk
    container_name: frigate
    privileged: true
    restart: unless-stopped
    ports:
      - 127.0.0.1:5000:5000 # Unsecure web interface used by home assistant, DO NOT EXPOSE
      - 127.0.0.1:8971:8971 # Secure web interface passed to NGINX
      - 0.0.0.0:8554:8554 # RTSP feeds
      - "0.0.0.0:8555:8555/tcp" # WebRTC over tcp
      - "0.0.0.0:8555:8555/udp" # WebRTC over udp
    shm_size: "111mb"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./media:/media/frigate
      - type: tmpfs # Optional: reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 64000000
    devices: # Since we run in privileged it's not required but fuck it
      - /dev/mpp_service:/dev/mpp_service
      - /dev/dri:/dev/dri
      - /dev/v4l:/dev/v4l
      - /dev/rga:/dev/rga
      - /dev/dma_heap:/dev/dma_heap
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/renderD129:/dev/dri/renderD129

Relevant Frigate log output

Since i restarted frigate, i can't read the logs of the outage is there a log history that i could copy from ?

Relevant go2rtc log output

-

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Sonoff RTSP cameras

Screenshots of the Frigate UI's System metrics pages

Couldn't reach them, the page would load forever. The only thing i could see is the bottom left status information saying "high CPU usage"

Any other information that may be helpful

No response

[inkeep-ai-assistant[bot]]

Your configuration uses the Rockchip NPU with the rknn detector and the deci-fp16-yolonas_m model, which is supported for Rockchip-based boards(1). High CPU and memory usage over time is not directly addressed in the documentation, but here are a few things you can check based on best practices from the sources:

• Ensure your shm_size is sufficient. For multiple cameras at 320x320 detection resolution, the documented calculation method is:

  $ python -c 'print("{:.2f}MB".format((<width> * <height> * 1.5 * 20 + 270480) / 1048576))'
  

  Add 40MB for logs and multiply by the number of cameras(2).

• If you see "Bus error" or crashes, low shm_size is often the cause(2).

• Using privileged mode is recommended during setup for hardware access, but once confirmed, consider restricting permissions for security(2).

• For memory or CPU leaks, the documentation suggests monitoring resource usage with docker stats, htop, iotop, and vmstat. Check for swapping, slow storage, or high I/O wait(3).

• If using network storage for media, NFS is recommended over SMB for performance reasons(3).

• If you only need detection and not continuous recording, adjust your recording or detection settings to reduce load(2).

There is no explicit log history feature for Frigate outages, but logs stored in Docker or your host (e.g., journalctl, docker logs frigate) are the main sources(2).

If you need further troubleshooting steps, you can review:

• Installation
• Hardware Acceleration
• Troubleshooting Recordings

📚 Sources:

1. Supported Hardware
2. Installation
3. Troubleshooting Recordings

---

Have another question? Just tag @inkeep.

[ShyneTurtle]

CPU usage graph as the system started becoming unresponsive:


Memory usage charts (note that i have ramfs running which progressively decreases the usable memory down to about 6.5GB, but it's not displayed on these charts)

[NickM-27]

A good place to start would be using tools like top inside of the container to see what is using memory and cpu, so that we can narrow down what process may be causing this behavior

[ShyneTurtle]

here is the result of top after letting the instance run for a while (it went up from 1.7GB to 2GB of memory use in about 30minutes)

[NickM-27]

it looks like the primary memory user is the embeddings process, which would be ONNXruntime for the semantic search models. This can depend on activity and on timing of when objects are detected and any searches are performed in the UI as the models are not initialized until that happens, and ONNXruntime dynamically adjusts how much memory it holds based on the size of the inputs.

Looks like for CPU the main user, which is still pretty low, is a camera and the associated processes. What detect resolutions are being run for these cameras?

[ShyneTurtle]

for now everything is pretty nominal, although 2.1GB on my 8GB server is a lot.

The cameras detection is persons only and run on the 3 NPU cores. I have motion masks for the datecode which is overlayed on the video feed, and on stuff that moves with the wind. You can see more info in my config.

[ShyneTurtle]

The AI model is a rockship-specific model (deci-fp16-yolonas_m) so it can run on the NPU

[NickM-27]

The AI model is a rockship-specific model (deci-fp16-yolonas_m) so it can run on the NPU

that is for object detection, the semantic search model runs on the CPU (or supported GPUs)

[ShyneTurtle]

After 2days of runtime, the memory is up to 2.77GB (almost 40% of the system memory). And the available system memory is now under 200MB.

Here is the memory usage inside the container:

[NickM-27]

Looks like the detector increased considerably along with what looks to be the main process. It's possible there's something incorrectly keeping memory in the RKNN python plugin, I'm not super familiar with how that works.

Realistically, we'll want to see a memray profile so we can know exactly what is using memory https://github.com/bloomberg/memray

[ShyneTurtle]

I'm not familiar with memray or "advanced" python programming in general, could you walk me through getting the data you need for debugging ?

[NickM-27]

it is somewhat advanced, you'd need to make adjustments to the docker container. Otherwise, we may need to see if other users on the same hardware report similar issues. I will also get my OrangePi running for an extended test and see what happens with the memory usage

[ShyneTurtle]

Okay, thank you !
If you need my help just @ me here !

[ShyneTurtle]

Another update this time with quite significant memory usage increase. The frigate instance is up to 4GB of memory usage (50% of system memory).

[ShyneTurtle]

Any updates on this or things i could do to gather data for you ?
Would rolling back to a previous version be okay ?

[NickM-27]

We haven't had any other reports, I haven't reproduced it

You definitely can rollback, you'll need to restore the backup config and database

[ShyneTurtle]

what about using memray ? can i do that inside the container ?

[NickM-27]

yes, you can use memray