[HW Accel Support]: Frigate container crashes, consuming all memory and CPU. #12186

bpbastos · 2024-06-28T12:56:17Z

bpbastos
Jun 28, 2024

Describe the problem you are having

I am experiencing an issue when enabling hardware acceleration in ffmpeg while running Frigate. The container running Frigate after some time crashes, consuming all memory and CPU. However, if I disable hardware acceleration, Frigate runs normally.

Version

0.13.2-6476F8A

Frigate config file

ui:
  live_mode: webrtc
  timezone: America/Sao_Paulo
  strftime_fmt: "%d/%m/%Y %H:%M:%S"


mqtt:
  host: host
  user: user
  password: pass

ffmpeg:
  hwaccel_args: preset-nvidia-h264

go2rtc:
  webrtc:
    listen: ":8555"
    candidates:
      - 192.168.40.168:8555
      - stun:8555
  streams:
    baby:
      - rtsp://admin:[email protected]:554/Streaming/Channels/1?transportmode=unicast&profile=Profile_1
      - "ffmpeg:baby#audio=opus"
    baby_sub:
      - rtsp://admin:[email protected]:554/Streaming/Channels/2?transportmode=unicast&profile=Profile_2

record:
  events:
    pre_capture: 7
    post_capture: 10

cameras:
  baby:
    ffmpeg:
      input_args: -rtsp_transport tcp
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/baby
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/baby_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio

    motion:
      threshold: 80
      lightning_threshold: 0.8
 #     contour_area: 10
      mask:
        - 0,0,512,0,512,221,446,88,304,62,163,116,0,194

    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.5
          threshold: 0.7
          mask: 
            - 0,0,512,0,512,221,446,88,304,62,163,116,0,194

    audio:
      enabled: true
      max_not_heard: 30
      min_volume: 30
      listen:
        - crying

    detect:
      enabled: true
      width: 512
      height: 288
#      fps: 8

    snapshots:
      enabled: true
#      timestamp: true
#      bounding_box: true
      retain:
        default: 10

    record:
      enabled: true
#      retain:
#        days: 7
#        mode: motion
      events:
#        objects:
#          - person
        retain:
          default: 10
          mode: active_objects

birdseye:
  enabled: false
  mode: motion

#detectors:
#  cpu1:
#    type: cpu
#    num_threads: 3
#    model:
#      path: "/custom_model.tflite"
#  cpu2:
#    type: cpu
#    num_threads: 3

#detectors:
#  ov:
#    type: openvino
#    device: AUTO
#    model:
#      path: /openvino-model/ssdlite_mobilenet_v2.xml

#model:
#  width: 300
#  height: 300
#  input_tensor: nhwc
#  input_pixel_format: bgr
#  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

detectors:
  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /config/model_cache/tensorrt/yolov7-320.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 320
  height: 320

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
#    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "256mb" # update for your cameras based on calculation above

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/frigate/config:/config
      - /home/frigate/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000

    ports:
      - "5555:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

    environment:
      NVIDIA_VISIBLE_DEVICES: all
      NVIDIA_DRIVER_CAPABILITIES: compute,utility,video
      USE_FP16: False
      YOLO_MODELS: yolov7-320
      PLUS_API_KEY: 

    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Relevant log output

frigate  | 2024-06-28T01:53:42.252536000Z 2024-06-27 22:53:42.252296212  [2024-06-27 22:53:42] watchdog.baby                  INFO    : baby exceeded fps limit. Exiting ffmpeg...
frigate  | 2024-06-28T01:53:42.259808000Z 2024-06-27 22:53:42.252701515  [2024-06-27 22:53:42] watchdog.baby                  INFO    : Waiting for ffmpeg to exit gracefully...
frigate  | 2024-06-28T01:53:47.115215000Z 2024-06-27 22:53:47.114839858  [2024-06-27 22:53:47] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/[email protected]
frigate  | 2024-06-28T01:55:11.671092000Z 2024-06-27 22:54:25.317304396  192.168.40.1 - - [27/Jun/2024:22:54:24 -0300] "GET /ws HTTP/1.1" 101 1234256 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.2.100"
frigate  | 2024-06-28T01:55:29.126312000Z 2024-06-27 22:54:35.086885079  22:54:35.063 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 127.0.0.1:8554->127.0.0.1:43146: i/o timeout" url=ffmpeg:baby#audio=opus
frigate  | 2024-06-28T01:56:18.191734000Z 2024-06-27 22:55:01.289865982  192.168.40.1 - - [27/Jun/2024:22:55:00 -0300] "GET /ws HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.2.100"
frigate  | 2024-06-28T01:57:05.911316000Z 2024-06-27 22:55:04.966176602  2024/06/27 22:55:03 [error] 740#740: *3461 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.40.80, server: , request: "GET /api/stats HTTP/1.1", upstream: "http://127.0.0.1:5001stats", host: "192.168.40.168:5555"
frigate  | 2024-06-28T01:59:47.682680000Z 2024-06-27 22:55:35.119049401  22:55:35.090 ERR [exec] timeout url="exec:ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/baby?audio -c:a libopus -application:a lowdelay -frame_duration 20 -min_comp 0 -vn -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
frigate  | 2024-06-28T02:41:09.084415000Z 2024-06-27 22:55:36.577586175  192.168.40.1 - - [27/Jun/2024:22:55:36 -0300] "GET /ws HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.2.100"
frigate  | 2024-06-28T02:41:24.509949000Z 2024-06-27 22:56:09.067916060  2024/06/27 22:56:08 [error] 1011#1011: *4202 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.40.80, server: , request: "GET /api/stats HTTP/1.1", upstream: "http://127.0.0.1:5001stats", host: "192.168.40.168:5555"
frigate  | 2024-06-28T02:41:26.861248000Z 2024-06-27 22:56:12.299425768  192.168.40.1 - - [27/Jun/2024:22:56:12 -0300] "GET /ws HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.2.100"
frigate  | 2024-06-28T02:41:27.352649000Z 2024-06-27 22:56:36.467602479  22:56:36.436 ERR [exec] timeout url="exec:ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/baby?audio -c:a libopus -application:a lowdelay -frame_duration 20 -min_comp 0 -vn -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
frigate  | 2024-06-28T02:41:27.761150000Z 2024-06-27 22:56:49.372591046  192.168.40.1 - - [27/Jun/2024:22:56:48 -0300] "GET /ws HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.2.100"
frigate  | 2024-06-28T02:41:30.443362000Z 2024-06-27 22:57:10.822265978  2024/06/27 22:57:10 [error] 155#155: *4271 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.40.80, server: , request: "GET /api/stats HTTP/1.1", upstream: "http://127.0.0.1:5001stats", host: "192.168.40.168:5555"
frigate  | 2024-06-28T02:41:33.713293000Z 2024-06-27 22:57:24.345310187  192.168.40.1 - - [27/Jun/2024:22:57:24 -0300] "GET /ws HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.2.100"
frigate  | 2024-06-28T02:41:34.416272000Z 2024-06-27 22:57:37.828733748  22:57:37.789 ERR [exec] timeout url="exec:ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/baby?audio -c:a libopus -application:a lowdelay -frame_duration 20 -min_comp 0 -vn -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
frigate  | 2024-06-28T02:41:40.042315000Z 2024-06-27 22:58:07.595404327  192.168.40.1 - - [27/Jun/2024:22:58:02 -0300] "GET /ws HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" "192.168.2.100"
frigate  | 2024-06-28T02:41:44.741821000Z 2024-06-27 22:58:18.558121379  2024/06/27 22:58:15 [error] 196#196: *4384 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.40.80, server: , request: "GET /api/stats HTTP/1.1", upstream: "http://127.0.0.1:5001stats", host: "192.168.40.168:5555"
frigate  | 2024-06-28T02:41:52.568073000Z 2024-06-27 22:58:39.144808225  22:58:39.083 ERR [exec] timeout url="exec:ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/baby?audio -c:a libopus -application:a lowdelay -frame_duration 20 -min_comp 0 -vn -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"15/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":720,"width":1280},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}},{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"8/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":288,"width":512},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}}]

Operating system

Proxmox

Install method

Docker Compose

Network connection

Wired

Camera make and model

Intelbras (Dahua) IC3 Wifi

Any other information that may be helpful

No response

hawkeye217 · 2024-06-28T13:32:25Z

hawkeye217
Jun 28, 2024
Collaborator

This is likely a duplicate and is an issue with the Proxmox kernel: #11676 #11958 #11977 #11101

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[HW Accel Support]: Frigate container crashes, consuming all memory and CPU. #12186

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

[HW Accel Support]: Frigate container crashes, consuming all memory and CPU. #12186

Uh oh!

Uh oh!

bpbastos Jun 28, 2024

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant log output

FFprobe output from your camera

Operating system

Install method

Network connection

Camera make and model

Any other information that may be helpful

Replies: 1 comment

Uh oh!

hawkeye217 Jun 28, 2024 Collaborator

bpbastos
Jun 28, 2024

hawkeye217
Jun 28, 2024
Collaborator