[Support]: GPU usage 100% after adding just 1 more camera

[kamonishe]

Describe the problem you are having

Good day,
I was quite successfully running frigate with 5 Uniview cameras (IPC2128LE-ADF28KM-G) for couple of months, everything is working amazing, object detection spot on, integration with home assistant is flawless in my use case (when a person is detected outside lights are switched on). Usually the load is somewhere around 20% CPU and 5% GPU. Inference time is around 10ms using USB Coral. Recently I've decided to add 2 more cameras and some weird thing happened. Upon adding 6th camera everything was fine, but after adding 7th camera GPU usage spiked to 94-100% and is staying that way until I disconnect 1 of the cameras (it can be any camera).
The machine I'm using as NVR is an old PC from more than a decade ago with following specs:
i7-2600
Radeon 6950
16GB RAM
I have already stocked up on cameras for the moment and I have 11 of them, all same Uniview model. It might happen that I will expand them to 20+ overall.
As far as I understand while looking at radeontop output GPU is the bottleneck. And I'm trying to understand if the hardware itself is the problem and I need to upgrade or there is a flaw in my config that is causing this and it is still possible to squeeze some more from this setup.

Thank you in advance.

Version

0.15-1

Frigate config file

mqtt:
  host: 192.168.0.2
  user: user
  password: password

ffmpeg:
  hwaccel_args: preset-vaapi

detectors:
  coral:
    type: edgetpu
    device: usb

go2rtc:
  streams:
    camera1: rtsp://192.168.0.101:554/media/video2
    camera2: rtsp://192.168.0.102:554/media/video2
    camera3: rtsp://192.168.0.103:554/media/video2
    camera4: rtsp://192.168.0.104:554/media/video2
    camera5: rtsp://192.168.0.105:554/media/video2
    camera6: rtsp://192.168.0.106:554/media/video2
    camera7: rtsp://192.168.0.107:554/media/video2
  webrtc:
    candidates:
      - 192.168.0.3:8555
      - stun

cameras:
  camera1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.101:554/media/video1
          hwaccel_args: preset-vaapi
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://192.168.0.101:554/media/video2
          hwaccel_args: preset-vaapi
          roles:
            - detect

    motion:
      mask: 
        0,0.54,0.218,0.286,0.31,0.239,0.321,0.228,0.326,0.198,0.518,0.082,0.562,0.088,0.578,0.096,0.576,0.151,0.724,0.167,0.821,0.183,0.951,0.213,0.957,0.198,1,0.207,1,0,0,0
      threshold: 50
      contour_area: 20
      improve_contrast: 'true'
    objects:
      filters:
        car:
          mask: 
            0.959,0.38,0.599,0.283,0.544,0.35,0.501,0.453,0.434,0.549,0.42,0.369,0.409,0.226,0.516,0.169,0.582,0.152,0.684,0.163,0.844,0.182,0.929,0.211,0.963,0.22

        person:
          mask: 0.326,0.416,0.384,0.358,0.407,0.59,0.359,0.7

  camera2:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.102:554/media/video1
          hwaccel_args: preset-vaapi
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://192.168.0.102:554/media/video2
          hwaccel_args: preset-vaapi
          roles:
            - detect

    motion:
      mask: 
        0,0.506,0,0,1,0,1,0.399,0.895,0.308,0.738,0.209,0.603,0.148,0.468,0.112,0.331,0.428
      threshold: 50
      contour_area: 20
      improve_contrast: 'true'
    objects:
      filters:
        car:
          mask: 
            0.516,0.284,0.433,0.208,0.629,0.236,0.706,0.262,0.808,0.384,1,0.561,1,0.869,0.597,0.366

  camera3:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.103:554/media/video1
          hwaccel_args: preset-vaapi
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://192.168.0.103:554/media/video2
          hwaccel_args: preset-vaapi
          roles:
            - detect

    motion:
      mask: 
        0.836,0.451,1,0.54,1,0,0,0,0,0.236,0.194,0.221,0.348,0.226,0.473,0.25,0.668,0.347,0.718,0.521

      threshold: 50
      contour_area: 20
      improve_contrast: 'true'
    objects:
      filters:
        car:
          mask: 0.077,0.227,0.076,0.024,0.698,0.119,1,0.3,0.999,0.539,0.484,0.247

  camera4:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.104:554/media/video1
          hwaccel_args: preset-vaapi
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://192.168.0.104:554/media/video2
          hwaccel_args: preset-vaapi
          roles:
            - detect

    motion:
      threshold: 50
      contour_area: 20
      improve_contrast: 'true'
      mask: 
        0,0.358,-0.001,0,1,0,1,0.166,0.769,0.076,0.566,0.073,0.381,0.087,0.384,0.158,0.224,0.206,0.118,0.241

    objects:
      filters:
        person:
          mask: 0.507,0.412,0.498,0.197,0.609,0.197,0.608,0.427

  camera5:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.105:554/media/video1
          hwaccel_args: preset-vaapi
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://192.168.0.105:554/media/video2
          hwaccel_args: preset-vaapi
          roles:
            - detect

    motion:
      threshold: 50
      contour_area: 20
      improve_contrast: 'true'
      mask: 0,0.524,0,0,1,0,1,0.072,0.841,0.067,0.631,0.139,0.361,0.258

  camera6:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.106:554/media/video1
          hwaccel_args: preset-vaapi
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://192.168.0.106:554/media/video2
          hwaccel_args: preset-vaapi
          roles:
            - detect

    motion:
      mask: 
        0,0.118,0.366,0.125,0.558,0.205,0.822,0.337,0.98,0.433,0.967,0.486,0.999,0.53,1,0.002,0,0.002
      threshold: 51
      contour_area: 20
      improve_contrast: true
    objects:
      filters:
        car:
          mask: 
            0.149,0.166,0.148,0.12,0.365,0.123,0.483,0.162,0.783,0.285,0.979,0.433,0.907,0.748
        person:
          mask: 
            0.144,0.175,0.145,0.125,0.36,0.125,0.614,0.204,0.789,0.282,0.99,0.416,0.903,0.755
  camera7:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.107:554/media/video1
          hwaccel_args: preset-vaapi
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://192.168.0.107:554/media/video2
          hwaccel_args: preset-vaapi
          roles:
            - detect

    motion:
      mask: 
        0,0.551,0,0,1,0,1,0.287,0.881,0.258,0.8,0.298,0.556,0.297,0.305,0.313,0.308,0.352
      threshold: 50
      contour_area: 20
      improve_contrast: true
detect:
  enabled: true
  width: 640
  height: 480
  fps: 5

objects:
  track:
    - person
    - car

record:
  enabled: true
  sync_recordings: false
  retain:
    days: 7
    mode: all
  alerts:
    retain:
      days: 30
  detections:
    retain:
      days: 30

snapshots:
  enabled: true
  retain:
    default: 30


logger:
  logs:
    frigate.record.maintainer: debug

version: 0.15-1
semantic_search:
  enabled: true
  reindex: false
  model_size: small

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "1024mb"
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/card0:/dev/dri/card0
      - /dev/bus/usb:/dev/bus/usb
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/docker/frigate:/config
      - /mnt/recordings:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      - LIBVA_DRIVER_NAME=radeonsi

Relevant Frigate log output

na

Relevant go2rtc log output

na

FFprobe output from your camera

na

Install method

Docker Compose

Object Detector

Coral

Network connection

Wired

Camera make and model

Uniview IPC2128LE-ADF28KM-G

Screenshots of the Frigate UI's System metrics pages

na

Any other information that may be helpful

No response

[ElectronicBattle]

I'm very much a novice but I have spotted two aspects which I wonder might be worth experimenting with:

(1)

go2rtc:
 streams:
  camera1: rtsp://192.168.0.101:554/media/video2
  camera2: rtsp://192.168.0.102:554/media/video2
  camera3: rtsp://192.168.0.103:554/media/video2
  camera4: rtsp://192.168.0.104:554/media/video2
  camera5: rtsp://192.168.0.105:554/media/video2
  camera6: rtsp://192.168.0.106:554/media/video2
  camera7: rtsp://192.168.0.107:554/media/video2

As far as I can see, you aren't making use of these streams elsewhere in your config, so what are you using go2rtc for, in that case? Perhaps you don't need it.

Also

(2) I presume video1 is the one on the left in the screenshot and video2 = the right hand . video2 is being used as the detect substream but you still have quite a high (1280x720) resolution chosen. Can you lower it to (say) 640x480? I think that might lower your GPU burden.

[NickM-27]

As far as I can see, you aren't making use of these streams elsewhere in your config, so what are you using go2rtc for, in that case? Perhaps you don't need it.

it would still be used for live views

[kamonishe]

Thanks for advice, I've tried fiddling with resolutions but it doesn't improve situation.

[NickM-27]

if GPU usage jumped that much from adding one camera it seems like that camera is sending a 4K stream which is overwhelming the GPU

[kamonishe]

The other thing I've noticed today is that during night time even with 7 cameras gpu usage was 5%, but as soon as sun came out it spiked back to 100%.

[NickM-27]

Less compression is possible during the day