[Detector Support]: D-FINE implementation with no Openvino hw acceleration

[brainstormi]

Describe the problem you are having

Initial implementation of d-fine seems to be working fine, kudos for such quick integration inside Frigate. Only issue detected is that when configuring onnx detector it doesn't seems to use Openvino gpu acceleration as Frigate docs suggests should be the case, multiplying by 10 the inference time compared to a similar complexity detector like yolonas M 640x640.
Aside from that would it be feasible to adjust the model to work with 320x320 resolution?.

Version

0.16#0de9287

Install method

Proxmox via Docker

Object Detector

OpenVino

Screenshots of the Frigate UI's System metrics pages

[NickM-27]

This is a WIP and we're not currently providing support for new features in the dev builds

[jamarju]

I was able to convert the ONNX models to OpenVINO following these instructions, which basically boil down to:

ovc dfine_m_obj2coco.onnx

Then I copied over the .xml + .bin files to /config/model_cache and configured the model and detectors in Frigate as follows:

detectors:
  ov_0:
    type: openvino
    device: GPU
  ov_1:
    type: openvino
    device: GPU

model:
  model_type: dfine
  width: 640
  height: 640
  input_tensor: nchw
  input_dtype: float
  path: /config/model_cache/dfine_m_obj2coco.xml
  labelmap_path: /labelmap/coco-80.txt

But the log shows this error for every camera:

frigate  | 2025-03-29 17:39:29.282352852  Traceback (most recent call last):
frigate  | 2025-03-29 17:39:29.282354141    File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
frigate  | 2025-03-29 17:39:29.282355055      self.run()
frigate  | 2025-03-29 17:39:29.282356045    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
frigate  | 2025-03-29 17:39:29.282356917      return run(*args, **kwargs)
frigate  | 2025-03-29 17:39:29.282357745             ^^^^^^^^^^^^^^^^^^^^
frigate  | 2025-03-29 17:39:29.282377729    File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
frigate  | 2025-03-29 17:39:29.282379471      self._target(*self._args, **self._kwargs)
frigate  | 2025-03-29 17:39:29.282380475    File "/opt/frigate/frigate/video.py", line 516, in track_camera
frigate  | 2025-03-29 17:39:29.282381323      process_frames(
frigate  | 2025-03-29 17:39:29.282382319    File "/opt/frigate/frigate/video.py", line 792, in process_frames
frigate  | 2025-03-29 17:39:29.282401427      detect(
frigate  | 2025-03-29 17:39:29.282402813    File "/opt/frigate/frigate/video.py", line 557, in detect
frigate  | 2025-03-29 17:39:29.282403884      region_detections = object_detector.detect(tensor_input)
frigate  | 2025-03-29 17:39:29.282404833                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
frigate  | 2025-03-29 17:39:29.282417660    File "/opt/frigate/frigate/object_detection.py", line 237, in detect
frigate  | 2025-03-29 17:39:29.282418844      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
frigate  | 2025-03-29 17:39:29.282419691       ~~~~~~~~~~~^^^^^^^^^^^
frigate  | 2025-03-29 17:39:29.282420451  KeyError: -959

[ReX1983]

You might want to train a custom model or use Frigate+. That inference time is not great. With even a modest amount of activity, you may end up with skipped detections.

Good point, it doesn't seem fit for my hardware.
I am just reading about Frigate+ and yolonas trained with user images, which should cover also foxes. This might be an option.

also, fox is not part of the COCO dataset so it not being detected as dog / cat is not a false negative, even if that is the desired behavior (a side effect of using a less accurate model). You could potentially get by with lowering the threshold for cat / dog

Right you, I got to admit that the low accuracy false positive ssdlite was definitely more helpful that yolonas not capturing them at all. I'll give a go at lowering threshold (and min_score?), but I am sure foxes won't show up when I need them for debugging reasons :-).

[NickM-27]

If you have a clip you can rerun it in frigate for debugging.

[ReX1983]

If you have a clip you can rerun it in frigate for debugging.

I had no idea this was possible. Any link explaining how to do that? I suppose the full video of the event, not just the screenshoot, right?

[hawkeye217]

See https://docs.frigate.video/development/contributing#2-modify-your-local-config-file-for-testing

[xavan83]

DFINE still doesn't work with the OpenVINO detector. It does work with the ONNX detector. It's a complex model, so even if it did work with the ONNX detector on iGPU, it would probably run too slowly.

Hello, I train a custom dfine nano model and export it to openvino.
The inference is 20msec for a 320*320size.
I try to export it in fp16 mode (model file size is reduced) but inference does not change.
Is it related to this limitation and that my model is running on the cpu?