[HW Accel Support]: RAM leaking with Nvidia, Unraid & Frigate

[DrSpaldo]

Describe the problem you are having

Recently I switched over to the latest version of Unraid and at the same time decided to try out hardware acceleration using a Nvidia 1050 in a friends server. The server totally crashed within 1 day. Eventually, after posting & receiving help from the Unraid forum, I was able to determine that the RAM from Frigate just continued to grow and consume the entire server.

After a bunch of tests, reviewing the logs and changing config settings, it looks like it is related to the Nvidia hardware acceleration.

The linked thread has some of the logs, further details and analysis from some of the guys over at Unraid.

Although I would like to reduce my CPU usage and move over some of the decoding to the Nvidia, I have just turned it off for the time being and reverted to CPU decoding. I have posted this if anyone developing the Frigate project is interested in having a look and also if others, like me, who read soooo many Unraid OOM posts find this and it may help.

Version

0.14.0-4e5a6eb

Frigate config file

mqtt:
  host: 192.168.2.2
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: unx
  password: yScxx
  stats_interval: 60
database:
  path: /db/frigate.db
ffmpeg:
  output_args:
    record: preset-record-generic-audio-copy
  hwaccel_args: preset-nvidia-h264
go2rtc:
  streams:
    front1:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    front1_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    front2:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    front2_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    garage1:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    garage1_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    garage2:
      - rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    garage2_sub:
      - rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1
    gate:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    gate_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    side1:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    side1_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    side2:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    side2_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    street1:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    street1_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    street2:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    street2_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    street3:
      - rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    street3_sub:
      - rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1
    back1:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=0#hardware
    back1_sub:
      - ffmpeg:rtsp://USERNAME:[email protected]:554/cam/realmonitor?channel=1&subtype=1#hardware
    back2:
      - rtsp://USERNAME:[email protected]:554/H264/ch1/main/av_stream
    back2_sub:
      - rtsp://USERNAME:[email protected]:554/H264/ch1/sub/av_stream
  webrtc:
    listen: :8555
    candidates:
      - 192.168.2.2:8555
      - stun:8555
cameras:
  front1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front1
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/front1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.7
      track:
        - person
    ui:
      order: 1
    birdseye:
      order: 1
      mode: continuous
    motion:
      mask:
        - 0,480,216,480,216,448,0,443
  front2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front2
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/front2_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.7
      track:
        - person
    ui:
      order: 2
    birdseye:
      enabled: false
    motion:
      mask:
        - 0,480,0,430,214,434,217,480
  gate:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gate
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/gate_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.7
      track:
        - person
    ui:
      order: 2
    birdseye:
      order: 2
      mode: objects
  garage1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage1
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/garage1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.75
          threshold: 0.8
      track:
        - person
        - car
    ui:
      order: 5
    birdseye:
      order: 3
      mode: objects
    motion:
      mask:
        - 210,440,208,480,110,480,106,444
  garage2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage2
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/garage2_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
      output_args:
        record: preset-record-generic
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.7
          threshold: 0.75
      track:
        - person
        - car
    ui:
      order: 6
    birdseye:
      enabled: false
    audio:
      enabled: false
    motion:
      mask:
        - 124,23,114,0,320,0,323,25
  back1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/back1
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/back1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.7
          threshold: 0.75
      track:
        - person
    ui:
      order: 7
    birdseye:
      order: 4
      mode: objects
    motion:
      mask:
        - 704,435,704,480,585,480,584,442
  back2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/back2
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
          hwaccel_args: preset-vaapi
        - path: rtsp://127.0.0.1:8554/back2_sub
          input_args: preset-rtsp-restream
          hwaccel_args: preset-vaapi
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 640
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.7
          threshold: 0.75
      track:
        - person
    ui:
      order: 8
    birdseye:
      enabled: false
    motion:
      mask:
        - 183,459,189,431,102,434,105,459
  side1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/side1
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/side1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 480
      height: 704
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.7
          threshold: 0.75
      track:
        - person
    ui:
      order: 52
    birdseye:
      order: 6
      mode: objects
    motion:
      mask:
        - 445,23,447,55,301,56,300,21
  side2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/side2
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/side2_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 15
          mode: active_objects
          objects:
            person: 30
    detect:
      enabled: true
      max_disappeared: 100
      width: 480
      height: 704
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    objects:
      filters:
        person:
          min_score: 0.7
          threshold: 0.75
      track:
        - person
    ui:
      order: 51
    birdseye:
      order: 7
      mode: objects
    motion:
      mask:
        - 268,0,267,42,139,46,135,0
  street1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/street1
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/street1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 10
          mode: active_objects
          objects:
            person: 14
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    ui:
      order: 3
    birdseye:
      order: 5
      mode: objects
    objects:
      track:
        - person
      filters:
        person:
          min_area: 350
          max_area: 16000
          min_score: 0.7
    motion:
      mask:
        - 704,23,704,0,569,0,571,21
    zones:
      street1_road:
        coordinates: 0.002,0.279,0.006,0.864,0.865,0.097,0.601,0.011,0.424,0.02
        objects: person
      street1_path:
        coordinates: 0.009,0.871,0.869,0.11,0.995,0.175,0.962,0.595,0.851,0.989,0.008,0.988
        objects: person
    review:
      detections:
        required_zones: street1_road
      alerts:
        required_zones: street1_path
  street2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/street2
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/street2_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 10
          mode: active_objects
          objects:
            person: 14
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 480
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
        objects:
          person: 60
    ui:
      order: 4
    objects:
      track:
        - person
      filters:
        person:
          min_area: 350
          max_area: 16000
          min_score: 0.7
          mask: 0.865,0.075,0.856,0.13,0.873,0.139,0.883,0.078
    motion:
      mask:
        - 203,0,204,26,103,26,100,0
    zones:
      street2_road:
        coordinates: 0.172,0.087,0.99,0.74,0.991,0.145,0.702,0.005,0.317,0.008
        objects: person
      street2_path:
        coordinates: 0.006,0.188,0.165,0.095,0.988,0.776,0.993,0.988,0.183,0.993,0.057,0.601
        objects: person
    review:
      detections:
        required_zones: street2_road
      alerts:
        required_zones: street2_path
  street3:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/street3
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/street3_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 10
        post_capture: 15
        objects:
          - person
        retain:
          default: 10
          mode: active_objects
          objects:
            person: 14
    detect:
      enabled: true
      max_disappeared: 100
      width: 704
      height: 576
      fps: 10
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      quality: 100
      crop: false
      retain:
        default: 7
    objects:
      track:
        - person
        - car
    ui:
      order: 12
    birdseye:
      enabled: false
    audio:
      enabled: false
    review:
      detections:
        labels:
          - person
      alerts:
        labels: []
    motion:
      threshold: 29
      contour_area: 11
      improve_contrast: 'false'
      lightning_threshold: 0.8
telemetry:
  network_interfaces: []
  stats:
    network_bandwidth: true

detectors:
  coral:
    type: edgetpu
    device: pci

birdseye:
  enabled: true
  restream: true
  width: 2592
  height: 1944
  quality: 1
  layout:
    max_cameras: 1
    scaling_factor: 1.0
camera_groups:
  Overview:
    order: 1
    icon: LuSmile
    cameras:
      - front1
      - garage1
      - street1
      - back1
      - front2
      - street2
      - gate
  Birdseye:
    order: 2
    icon: LuBird
    cameras: birdseye
  Home_Assistant:
    order: 3
    icon: LuHome
    cameras:
      - back1
      - front1
      - garage1
      - gate
      - street1
      - back2
review:
  alerts:
    labels:
      - person
  detections:
    labels:
      - yell
audio:
  enabled: true
  min_volume: 600
  listen:
    - bark
    - scream
    - speech
    - yell
version: 0.14

docker-compose file or Docker CLI command

Unraid. With --gpus=all

Relevant log output

See linked thread

FFprobe output from your camera

Can get if needed

Operating system

Proxmox

Install method

Docker Compose

Network connection

Wired

Camera make and model

Amcrest

Any other information that may be helpful

Nvidia 1050

[DrSpaldo]

Here are some of the related posts I found along the way:

#5976 (comment) : Only wyze cams - user explains different scenarios that work and wyze + go2rtc is the problem. Upgrading go2rtc fixed the issue

#5924 : Only wyze cams + go2rtc - ongoing

#6645 (comment) : Many cameras including wyze - upgrading go2rtc fixed the issue

#6715 (comment) : Some reolink & some amcrest - upgrading go2rtc fixed the issue

#6174 (comment) : All reolink cameras - docker config issue caused the container (not ffmpeg process) to use large amount of memory

#6469 (comment) : All reolink cameras - ended up being unrelated to frigate at all

#5799 : Different Cameras - issue caused by bad kernel on host OS, updating fixed issue

#6731

#7476

[DrSpaldo]

This is the Frigate app/container/docker RAM usage AFTER reverting from Nvidia HW to CPU HW:

19/6 10.30am - 4.11g
19/6 11.15am - 4.08g
19/6 2.00pm - 4.13g
19/6 4.15pm - 4.4g
19/6 8.45pm - 4.6g
20/6 8.15am - 4.7g
20/6 8.30pm - 4.9g
21/6 1.00pm - 6.1g
21/6 10.00pm - 5.8g

[capnjip]

I have a similar issue, with the OOM warnings. I am running Frigate on Unraid, with GTX1060 for video decoding, I am writing in hopes for a resolution to this issue.