[HW Accel Support]: High CPU Usage with HW Acceleration

[strawgate]

Describe the problem you are having

Hi,

I am running the following config on a OrangePi 5 Plus with a rockchip that has rknn npu and rkmpp h264 decoder

mqtt:
  enabled: false

detectors:
  rknn:
    type: rknn
    core_mask: 0b011

database:
  path: /config/frigate.db
  
model:
  input_pixel_format: bgr

audio:
  enabled: True
  listen:
    - bark
    - fire_alarm
    - scream
    - speech
    - sing
    - television
    - yell
    - crow
    - christian_music
    - christmas_music
    - angry_music
    - scary_music
    - sizzle
    - smoke_detector
    - fire_alarm

objects:
  track:
    - person
    - bird
    - crow
    - motorcycle
    - airplane
    - bus
    - dog
    - apple
    - spoon
    - knife
    - fork
    - cup
    - bicycle

record:
  enabled: True
  retain:
    days: 1
snapshots:
  enabled: True
detect:
  enabled: True

go2rtc:
  streams:
    unifi-street-railing:
      - rtspx://192.168.0.1:7441/xxxxxxxxxxxxxxxxxxx

ffmpeg:
  input_args: preset-rtsp-restream
  hwaccel_args: preset-rk-h264

cameras:
  unifi-street-railing:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/unifi-street-railing
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-ubiquiti
    detect:
      width: 1280
      height: 720
    motion:
      threshold: 60
      mask:
        - 0,720,123,720,123,91,646,172,638,320,811,720,1003,720,1280,720,1280,0,0,0

Frigate is reporting 30% CPU Usage for the ffmpeg process with this configuration:

If I comment out hwaccel_args: preset-rk-h264, i see the same CPU usage in Frigate:

Any advice?

Version

0.13.2-6476F8A

Frigate config file

mqtt:
  enabled: false

detectors:
  rknn:
    type: rknn
    core_mask: 0b011

database:
  path: /config/frigate.db
  
model:
  input_pixel_format: bgr

audio:
  enabled: True
  listen:
    - bark
    - fire_alarm
    - scream
    - speech
    - sing
    - television
    - yell
    - crow
    - christian_music
    - christmas_music
    - angry_music
    - scary_music
    - sizzle
    - smoke_detector
    - fire_alarm

objects:
  track:
    - person
    - bird
    - crow
    - motorcycle
    - airplane
    - bus
    - dog
    - apple
    - spoon
    - knife
    - fork
    - cup
    - bicycle

record:
  enabled: True
  retain:
    days: 1
snapshots:
  enabled: True
detect:
  enabled: True

go2rtc:
  streams:
    unifi-street-railing:
      - rtspx://192.168.0.1:7441/xxxxxxxxxxxxxxxxxxx

ffmpeg:
  input_args: preset-rtsp-restream
  hwaccel_args: preset-rk-h264

cameras:
  unifi-street-railing:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/unifi-street-railing
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-ubiquiti
    detect:
      width: 1280
      height: 720
    motion:
      threshold: 60
      mask:
        - 0,720,123,720,123,91,646,172,638,320,811,720,1003,720,1280,720,1280,0,0,0

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate

    privileged: true # this may not be necessary for all setups
    restart: always

    image: ghcr.io/blakeblackshear/frigate:stable-rk
    shm_size: "96mb" # update for your cameras based on calculation above

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - frigate_config:/config
      - frigate_media:/media/frigate
      #- /media/frigate:/media/frigate:z
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/renderD129:/dev/dri/renderD129
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000

    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

    environment:
      FRIGATE_RTSP_PASSWORD: "password"

volumes:
  frigate_config:
  frigate_media:

Relevant log output

2024-04-04 18:43:41.043512244  [INFO] Preparing Frigate...
2024-04-04 18:43:41.083634480  [INFO] Starting Frigate...
2024-04-04 18:43:42.892160817  [2024-04-04 18:43:42] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)
2024-04-04 18:43:42.916894366  [2024-04-04 18:43:42] peewee_migrate.logs            INFO    : Starting migrations
2024-04-04 18:43:42.922933234  [2024-04-04 18:43:42] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-04-04 18:43:42.972480247  [2024-04-04 18:43:42] frigate.app                    INFO    : Recording process started: 450
2024-04-04 18:43:42.975187456  [2024-04-04 18:43:42] frigate.app                    INFO    : go2rtc process pid: 84
2024-04-04 18:43:43.002826545  [2024-04-04 18:43:43] frigate.app                    INFO    : Output process started: 460
2024-04-04 18:43:43.003317996  [2024-04-04 18:43:43] detector.rknn                  INFO    : Starting detection process: 459
2024-04-04 18:43:43.042944990  [2024-04-04 18:43:43] frigate.app                    INFO    : Camera processor started for unifi-street-railing: 466
2024-04-04 18:43:43.050352338  [2024-04-04 18:43:43] frigate.app                    INFO    : Capture process started for unifi-street-railing: 468
2024-04-04 18:43:43.052404474  [2024-04-04 18:43:43] frigate.app                    INFO    : Audio process started: 472
2024-04-04 18:43:43.282098764  I RKNN: [18:43:43.282] RKNN Runtime Information: librknnrt version: 1.5.2 (c6b7b351a@2023-08-23T15:28:22)
2024-04-04 18:43:43.282179846  I RKNN: [18:43:43.282] RKNN Driver Information: version: 0.9.2
2024-04-04 18:43:43.282556966  I RKNN: [18:43:43.282] RKNN Model Information: version: 6, toolkit version: 1.5.2+b642f30c(compiler version: 1.5.2 (c6b7b351a@2023-08-23T07:39:01)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"30/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":720,"width":1280},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}}]

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Unifi G5 Bullet

Any other information that may be helpful

No response

[strawgate]

Sounds like it's related to #10034 and I'll have to wait for the next version of Frigate?

[strawgate]

Running ghcr.io/blakeblackshear/frigate:dev-0223d6d-rk seems to drop the CPU to around 10%