OpenVino On AMD Ryzen CPU

[adarshahd]

Hello all,

First of all, thank you very much for a great project. I have a MinisForum UM790 pro (Ryzen 7945HS) mini pc, which has Radeon 780M gpu. Unfortunately I had to use CPU detectors as AMD doesn't have a proper SW for using GPU for this use case, although this chipset has inbuilt TPU (8 TOPS). So my inference speeds were anywhere around 30ms and was not happy

So today I found out we can use OpenVINO detectors for AMD CPU/GPUs also. And alas ! my inference speeds has dropped to 3ms !!

This is FYI all

[NickM-27]

what is your config?

[adarshahd]

Here it is, pretty basic. Just that using OpenVINO instead of CPU

mqtt:
  enabled: false
ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-aac
cameras:
  front:
    ffmpeg:
      inputs:
        - path: rtsp://***:***@192.168.1.5:554/stream1
          roles:
            - detect
    detect:
      width: 1920
      height: 1080
record:
  enabled: True
  retain:
    days: 7
    mode: motion
  events:
    retain:
      default: 14
      mode: active_objects
detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml
model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

[NickM-27]

If I had to guess it is still using the CPU (not the GPU), the only way to confirm for sure that it is using the GPU will be to change the device from AUTO to GPU

[adarshahd]

You are right. It would still use the CPU. We can see from the log that it fails if we change the config to use GPU.

RuntimeError: Failed to create plugin /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_gpu_plugin.so for device GPU

However even with CPU the performance gained is enormous from 30ms -> 3ms. I can also see CPU usage going down from previous values

[adarshahd]

I also updated the title accordingly :)

[adarshahd]

And docker-compose. I run Fedora and use podman

version: "3.9"
services:
  frigate:
    container_name: frigate
    restart: unless-stopped
#    privileged: true
    image: ghcr.io/blakeblackshear/frigate:0.13.0-beta6
    shm_size: "64mb" # update for your cameras based on calculation above
    volumes:
      - ./config/:/config/:Z
      - ./storage:/media/frigate:Z
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"
      NVIDIA_VISIBLE_DEVICES: void
      LIBVA_DRIVER_NAME: radeonsi

[GrumpyMeow]

I'm curious why the performance would improve using OpenVINO on a AMD Ryzen CPU.
Is it because of the model which is more efficient?
Is the "detector" more efficient?
Or is it using more efficient CPU features?

[NickM-27]

it is because the openvino detector is more efficient than running tflite directly on the cpu

[marcschumacher]

This might be a very basic question, but I read that OpenVINO was mostly created for Intel CPUs. Can you tell me how to install the drivers or is there any documentation available? Furthermore, can I assume that using the hardware acceleration itself also helps for decoding the ffmpeg streams?
I am currently using my MinisForum UM790 with Proxmox, including a Coral USB device, but I am having a lot of cameras and I would like to benefit from any additional performance my machine provides.

[ivanjx]

which intel cpu you were using before switching to the amd cpu? also what is the cpu usage while detecting an object on the amd cpu? thanks

[vishalrao88]

Thanks for sharing!

I was able to run openvino on a AMD 5800U (15W) with success! yolov8n: PT with ROCM -> 35-45 ms vs Openvino (amd cpu only) -> 25-35 ms.

I am interested in further optimizations. Have you had success with Rocm or other combos? e.g. RocM+Openvino. Anything thats faster than openvino (cpu only)?

[ivanjx]

what is the cpu usage currently?

[vishalrao88]

iirc about 70-75% while the code's running

[sharbich]

Hello,
i have installed Homeassistant and Frigate on a mini PC as a docker with the following CPU:
AMD Ryzen 5 5500U with Radeon Graphics
If I use OpenVINO in the Frigate.yml with the following content:

detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml
model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

I get the following error message:

2024-12-17 11:32:03.414588727  Process detector:ov:
2024-12-17 11:32:03.415900583  Traceback (most recent call last):
2024-12-17 11:32:03.415924261    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-12-17 11:32:03.415928032      self.run()
2024-12-17 11:32:03.415931315    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-12-17 11:32:03.415938928      self._target(*self._args, **self._kwargs)
2024-12-17 11:32:03.415942770    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-12-17 11:32:03.415961838      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-12-17 11:32:03.415965610    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2024-12-17 11:32:03.415968962      self.detect_api = create_detector(detector_config)
2024-12-17 11:32:03.415972315    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-12-17 11:32:03.415976157      return api(detector_config)
2024-12-17 11:32:03.415979579    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 43, in __init__
2024-12-17 11:32:03.415999974      self.interpreter = self.ov_core.compile_model(
2024-12-17 11:32:03.416003816    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 521, in compile_model
2024-12-17 11:32:03.416015760      super().compile_model(model, device_name, {} if config is None else config),
2024-12-17 11:32:03.416034269  RuntimeError: Exception from src/inference/src/cpp/core.cpp:123:
2024-12-17 11:32:03.416037691  Exception from src/inference/src/dev/plugin.cpp:59:
2024-12-17 11:32:03.416041114  Check 'contexts.count(device_id)' failed at src/plugins/intel_gpu/src/plugin/plugin.cpp:204:
2024-12-17 11:32:03.416043000  [GPU] Context was not initialized for 0 device
2024-12-17 11:32:03.416046352 

What's wrong with my configuration?
Greetings from Stefan

[NickM-27]

You need to set the mode to CPU not AUTO

[JoshC1994]

any update on the NPU front? is this something that could be supported with the AMD cpu's? https://www.minisforum.com/pages/n5_pro

[sharbich]

Hello NickM-27,
Thank you very much for your advice. It works now. Can I ask you another question? After about 1 - 2 days I have error messages in the ffmpeg process in Frigate. See error messages:

error 2024-12-18 07:13:41 ffmpeg.reolink.detect rtsp://*:*@192.168.130.143:554/h264Preview_01_sub: No route to host
error 2024-12-18 07:13:44 frigate.video reolink: Unable to read frames from ffmpeg process.
error 2024-12-18 07:13:44 frigate.video reolink: ffmpeg process is not running. exiting capture thread...
error 2024-12-18 07:13:51 watchdog.reolink Ffmpeg process crashed unexpectedly for reolink.
error 2024-12-18 07:13:51 watchdog.reolink The following ffmpeg logs include the last 100 lines prior to exit.
error 2024-12-18 07:13:51 ffmpeg.reolink.detect [tcp @ 0x55a40f0782c0] Connection to tcp://192.168.130.143:554?timeout=5000000 failed: No route to host
error 2024-12-18 07:13:51 ffmpeg.reolink.detect rtsp://*:*@192.168.130.143:554/h264Preview_01_sub: No route to host

How do I stop the error messages?
Greetings from Stefan

[NickM-27]

No route to host means the camera is offline or changed ip addresses