[HW Accel Support]: Inappropriate addition of FPS and Scale to detect role FFMPEG command

[foxale08]

Describe the problem you are having

I am attempting to modify the hardware offload parameters for the detect role of one of my cameras to better optimize CPU and RKRGA resources but have run across a snag. There doesn't appear to be a way to remove the additional string of "-r 5 -vf fps=5,scale=3840:2160" which is added to the command. I am aware of a similar issue #4093 which mentions this string but I need the string removed, not the detect process.

The long winded backstory is as follows. The hardware in question is a CM3588 NAS board which pretty much requires hardware offloading to be useful. I use 4K cameras which support 4K/HEVC on the main stream and 1080P/AVC on the secondary. Through experimentation I determined that my object detection is significantly better with a 4K image than at 1080p due to the deep depth of field of my cameras. The caveat with the "preset-rk-h265" is that when use for the detector role it hammers the single "rga2" scheduler. In my case four 4K detect streams push the rga2 scheduler over 60% load during the day which is problematic because I'd like to have eight cameras, I have plenty of CPU and NPU resources left (1x hailo8.) I am attempting to more efficiently use the rga2 scheduler or at the very least push more load towards the rga3 schedulers and CPU but not too much as the conversion easily uses a core and a half without hardware.

cameras:
CAM-DWNE:
audio:
enabled: true
record:
retain:
days: 14
mode: all
ffmpeg:
inputs:
- path: rtsp://go2rtc:[email protected]:8554/CAM_DWNE
hwaccel_args: preset-rk-h265
roles:
- audio
- record
- path: rtsp://go2rtc:[email protected]:8554/CAM_DWNE
input_args: -hide_banner -loglevel warning -rtsp_transport tcp -timeout 5000000
hwaccel_args: -hwaccel rkmpp -hwaccel_output_format drm_prime
roles:
- detect
output_args:
record: preset-record-generic-audio-copy
detect: -vf fps=5,scale_rkrga=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -f rawvideo -pix_fmt yuv420p

As seen from "cat /sys/kernel/debug/rkrga/load"

num of scheduler = 3
================= load ==================
scheduler[0]: rga3
load = 29%

scheduler[1]: rga3
load = 8%

scheduler[2]: rga2
load = 39%

........
747 active 406614 /usr/lib/ffmpeg/ffmpeg-jellyfin//bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -hide_banner -loglevel warning -rtsp_transport tcp -timeout 5000000 -i rtsp://go2rtc:[email protected]
.1:8554/CAM_DWNE -r 5 -vf fps=5,scale=3840:2160 -vf fps=5,scale_rkrga=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -f rawvideo -pix_fmt yuv420p pipe:
.......

Version

0.15

Frigate config file

mqtt:
  enabled: false

detectors:
  hailo8:
    type: hailo8l
    device: PCIe

ffmpeg:
  path: /usr/lib/ffmpeg/ffmpeg-jellyfin/

model:
  width: 640
  height: 640
  input_tensor: nhwc
  input_pixel_format: rgb
  path: /config/model_cache/h8l_cache/yolov8s.hef
  model_type: yolox
  labelmap_path: /labelmap.txt

motion:
  enabled: true

snapshots:
  enabled: false
  timestamp: true
  bounding_box: true
  retain:
    default: 0.5

record:
  enabled: true
  retain:
    days: 7
    mode: all
  alerts:
    pre_capture: 30
    post_capture: 30
    retain:
      days: 21
      #mode: active_objects
  detections:
    pre_capture: 30
    post_capture: 30
    retain:
      days: 0.5
      #mode: active_objects

birdseye:
  enabled: true
  mode: continuous
  restream: false
  height: 2160
  width: 3840

go2rtc:
  rtsp:
    listen: :8554
    username: go2rtc
    password: XXXXXXXXXXX
  streams:

    CAM_DWNE:
      - exec:/usr/lib/ffmpeg/7.0/bin/ffmpeg -hwaccel rkmpp -hwaccel_output_format
        drm_prime -extra_hw_frames 5 -afbc rga -rtsp_transport tcp -timeout 5000000
        -i rtsp://go2rtc:[email protected]:8554/CAM_DWNE -c:a copy -strict
        -2 -vf fps=20,scale_rkrga=w=3840:h=2160:format=nv12:afbc=1 -c:v hevc_rkmpp
        -rc_mode VBR -b:v 3M -maxrate 12M -bufsize 12M -profile:v main -level 5.1
        -g:v 100 -f rtsp {{output}}
........
    CAM_DWNE_LIVE:
      - exec:/usr/lib/ffmpeg/7.0/bin/ffmpeg -hwaccel rkmpp -hwaccel_output_format
        drm_prime -afbc rga -rtsp_transport tcp -timeout 5000000 -i rtsp://go2rtc:[email protected]:8554/CAM_DWNE
        -c:a copy -strict -2 -vf fps=20,scale_rkrga=w=3840:h=2160:format=nv12:afbc=1
        -c:v h264_rkmpp -rc_mode VBR -b:v 6M -maxrate 6M -bufsize 12M -profile:v high
        -g:v 120 -f rtsp {{output}}
.......
    CAM_DWNE_DETECT:
      - exec:/usr/lib/ffmpeg/7.0/bin/ffmpeg -hwaccel rkmpp -hwaccel_output_format
        drm_prime -afbc rga -rtsp_transport tcp -timeout 50000000 -i rtsp://go2rtc:[email protected]:8554/CAM_DWNE
        -c:a copy -strict -2 -vf fps=5,scale_rkrga=w=3840:h=2160:format=nv12:afbc=1
        -c:v h264_rkmpp -rc_mode VBR -b:v 2M -maxrate 4M -bufsize 12M -profile:v high
        -g:v 120 -f rtsp {{output}}
.......
cameras:
  CAM-DWNE:
    audio:
      enabled: true
    record:
      retain:
        days: 14
        mode: all
    ffmpeg:
      inputs:
        - path: rtsp://go2rtc:[email protected]:8554/CAM_DWNE
          hwaccel_args: preset-rk-h265
          roles:
            - audio
            - record
        - path: rtsp://go2rtc:[email protected]:8554/CAM_DWNE
          input_args: -hide_banner -loglevel warning -rtsp_transport tcp -timeout 5000000
          hwaccel_args: -hwaccel rkmpp -hwaccel_output_format drm_prime
          roles:
            - detect
      output_args:
        record: preset-record-generic-audio-copy
        detect: -vf fps=5,scale_rkrga=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -f rawvideo -pix_fmt yuv420p

    live:
      stream_name: CAM_DWNE_LIVE
    detect:
      fps: 5
      enabled: true
      width: 3840
      height: 2160
      min_initialized: 5
      max_disappeared: 1500
      stationary:
        interval: 25
        threshold: 50
    objects:
      track:
        - person
        - car
        - bicycle
      filters:
        car:
          mask:
            - 0.003,0.391,0.11,0.296,0.09,0.229,0,0.31
            - 0.456,0.041,0.614,0.01,0.675,0.036,0.648,0.063,0.523,0.086
            - 0.271,0.997,0.842,1,0.981,0.437,0.485,0.213,0.121,0.456
            - 0.717,0.16,0.967,0.299,0.979,0.177,0.976,0.12,0.935,0.1,0.89,0.139,0.853,0.148,0.82,0.135,0.768,0.141
            - 0.747,0.046,0.831,0.067,0.834,0.029,0.748,0.012
    motion:
      mask: 0,0,0.298,0,0.298,0.031,0,0.03
      threshold: 35
      contour_area: 50
....
version: 0.15-1
semantic_search:
  enabled: true
  reindex: false
  model_size: small

docker-compose file or Docker CLI command

services:
  frigate:
    #image: ghcr.io/blakeblackshear/frigate:0.13.2-rk
    #image: ghcr.io/blakeblackshear/frigate:stable-rk
    image: ghcr.io/blakeblackshear/frigate:stable-h8l
    privileged: true
    container_name: frigate
    hostname: frigate
    restart: unless-stopped
    shm_size: "4096mb"
    devices:
      - /dev/video-dec0:/dev/video-dec0
      - /dev/video-enc0:/dev/video-enc0
      - /dev/dri
      - /dev/dma_heap
      - /dev/rga
      - /dev/mpp_service
      - /dev/hailo0
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /frigate/config:/config
      - /frigate/media:/media/frigate
      - /frigate/ffmpeg/ffmpeg-marka711/ffmpeg:/usr/lib/ffmpeg/6.0/bin/ffmpeg
      - /frigate/ffmpeg/ffmpeg-jellyfin/ffmpeg:/usr/lib/ffmpeg/7.0/bin/ffmpeg
      - /frigate/ffmpeg/ffmpeg-marka711/ffmpeg:/usr/lib/ffmpeg/ffmpeg-marka711/bin/ffmpeg
      - /frigate/ffmpeg/ffmpeg-jellyfin/ffmpeg:/usr/lib/ffmpeg/ffmpeg-jellyfin/bin/ffmpeg
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      - FRIGATE_RTSP_USER=nvr1
      - FRIGATE_RTSP_PASSWORD=XXXXXXXXXXXXXXXXXXXX
      - FRIGATE_MQTT_HOST=127.0.0.1
      - FRIGATE_MQTT_USER=nvr3
      - FRIGATE_MQTT_PASSWORD=XXXXXXXXXXXXXXXXXXXX

networks:
  default:
    driver: bridge
    driver_opts:
      com.docker.network.driver.mtu: 1450

Relevant Frigate log output

N/A

Relevant go2rtc log output

N/A

FFprobe output from your camera

N/A

Install method

Docker Compose

Object Detector

Other

Network connection

Wired

Camera make and model

Assorted 4K Hikvision

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[hawkeye217]

Your issue is coming from your use of preset-rk-h265. You'll need to manually specify hwaccel_args if you want to remove the vf added there.

See frigate/ffmpeg_presets.py line 98.

[foxale08]

Appreciate the quick response! The string is getting added by default, even when hwaccel_args is manually specified. I experimented with a single ffmpeg/multiple ffmpeg processes and if a listed preset isn't specified the default is added in addition to the manual hwaccel_args string.

I'm not an expert but wouldn't putting the default scaling options in default output_args section of schema.json allow for a blank default "PRESETS_HW_ACCEL_SCALE" in ffmpeg_presets.py? The options appear to already partially exist in schema.json.

[hawkeye217]

@NickM-27 may have more thoughts here as he wrote that particular part of the code.

[NickM-27]

In general this is working as expected, due to the requirements of how Frigate works.

you could adjust the ffmpeg_presets.py to add a new preset with the args you want