Which Frigate+ model to use with NPU #19273
Replies: 4 comments 15 replies
-
|
For Frigate+ models on Intel NPUs via OpenVINO, the recommended model type is yolonas(1). The documentation specifies for Intel hardware (including platforms supporting OpenVINO) that you should use the openvino detector type with yolonas model type for Frigate+(1). Your detector config should look like this: detectors:
ov:
type: openvino
device: NPU
model:
model_type: yolonas
width: 320 # or 640, must match the model export
height: 320 # or 640, must match the model export
input_tensor: nchw
input_pixel_format: bgr
path: /config/yolo_nas_s.onnx # or the downloaded Frigate+ yolonas model path
labelmap_path: /labelmap/coco-80.txtMake sure you are using the yolonas variant from Frigate+ and that the model is exported in a format compatible with OpenVINO. The mobiledet model type is not recommended or supported for OpenVINO/Intel NPUs(1). Reference: 📚 Sources: Have another question? Just tag @inkeep. |
Beta Was this translation helpful? Give feedback.
-
|
For the record, this doesn't work. I get the same error. |
Beta Was this translation helpful? Give feedback.
-
|
There is no official NPU support, I have only seen a couple reports that YOLO-NAS was working with NPU and the performance was considerably worse than GPU. In general, using GPU is recommended as the NPU is generally not powerful enough to run a larger model like YOLO-NAS |
Beta Was this translation helpful? Give feedback.
-
|
No, they are tflite format and not compatible with openvino in general You may see if yolov9 works, as that will be available for frigate+ in the future |
Beta Was this translation helpful? Give feedback.
-
|
Define "future" 😅 🔜 ? |
Beta Was this translation helpful? Give feedback.
-
|
Blake is still working on it and there is no timeframe available. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, guys! |
Beta Was this translation helpful? Give feedback.
-
|
Blake said the goal is before the October base model, assuming no major issues are hit |
Beta Was this translation helpful? Give feedback.
-
|
I'm success used intel NPU (Intel Core Ultra 7 258v) on Frigate with YOLO-NAS. NPU don't run on flatten export that same problem run YOLO-NAS on Rockchip NPU. I modify openvino.py for support this model. |
Beta Was this translation helpful? Give feedback.
-
|
Huh, that's very encouraging, what changes did you need to make to openvino.py? I'm assuming that was accompanied by exporting the model in non-flat format |
Beta Was this translation helpful? Give feedback.
-
|
Below code support my custom converter tools or Your tools add postprocessing=False. (Remove NMS) import logging
import os
import numpy as np
import openvino as ov
from pydantic import Field
from typing_extensions import Literal
from frigate.detectors.detection_api import DetectionApi
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
from frigate.util.model import (
post_process_dfine,
post_process_rfdetr,
post_process_yolo,
)
from frigate.util.builtin import load_labels
logger = logging.getLogger(__name__)
DETECTOR_KEY = "openvino"
class OvDetectorConfig(BaseDetectorConfig):
type: Literal[DETECTOR_KEY]
device: str = Field(default=None, title="Device Type")
class OvDetector(DetectionApi):
type_key = DETECTOR_KEY
supported_models = [
ModelTypeEnum.dfine,
ModelTypeEnum.rfdetr,
ModelTypeEnum.ssd,
ModelTypeEnum.yolonas,
ModelTypeEnum.yologeneric,
ModelTypeEnum.yolox,
]
def __init__(self, detector_config: OvDetectorConfig):
super().__init__(detector_config)
self.ov_core = ov.Core()
self.ov_model_type = detector_config.model.model_type
self.h = detector_config.model.height
self.w = detector_config.model.width
# self.thresh = 0.3 # Default threshold for filtering detections
self.labelmap_path = detector_config.model.labelmap_path
self.num_label = len(load_labels(self.labelmap_path, "utf-8", 0) )
if not os.path.isfile(detector_config.model.path):
logger.error(f"OpenVino model file {detector_config.model.path} not found.")
raise FileNotFoundError
self.interpreter = self.ov_core.compile_model(
model=detector_config.model.path, device_name=detector_config.device
)
self.model_invalid = False
if self.ov_model_type not in self.supported_models:
logger.error(f"OpenVino detector does not support {self.ov_model_type} models.")
self.model_invalid = True
if self.ov_model_type == ModelTypeEnum.ssd:
model_inputs = self.interpreter.inputs
model_outputs = self.interpreter.outputs
if len(model_inputs) != 1 or len(model_outputs) != 1:
logger.error("SSD models must have exactly 1 input and 1 output.")
self.model_invalid = True
else:
output_shape = model_outputs[0].get_shape()
if output_shape[0] != 1 or output_shape[1] != 1 or output_shape[3] != 7:
logger.error(f"SSD model output shape mismatch: {output_shape}")
self.model_invalid = True
if self.ov_model_type == ModelTypeEnum.yolonas:
model_inputs = self.interpreter.inputs
model_outputs = self.interpreter.outputs
if len(model_inputs) != 1:
logger.error(f"YOLOโ€‘NAS models must have exactly 1 input. Found {len(model_inputs)}.")
self.model_invalid = True
self.raw_yolonas_mode = False
if len(model_outputs) == 2:
self.raw_yolonas_mode = True
logger.info("Detected YOLOโ€‘NAS raw mode (boxes + scores).")
elif len(model_outputs) == 1:
logger.info("Detected YOLOโ€‘NAS flat mode.")
output_shape = model_outputs[0].partial_shape
if output_shape[-1] != 7:
logger.error(f"YOLOโ€‘NAS flat model output shape mismatch: {output_shape}")
self.model_invalid = True
else:
logger.error(f"Unexpected number of YOLOโ€‘NAS outputs: {len(model_outputs)}.")
self.model_invalid = True
if self.ov_model_type == ModelTypeEnum.yolox:
self.output_indexes = 0
while True:
try:
tensor_shape = self.interpreter.output(self.output_indexes).shape
logger.info(f"Model Output-{self.output_indexes} Shape: {tensor_shape}")
self.output_indexes += 1
except Exception:
logger.info(f"Model has {self.output_indexes} Output Tensors")
break
self.num_classes = tensor_shape[2] - 5
logger.info(f"YOLOX model has {self.num_classes} classes")
self.calculate_grids_strides()
def process_yolo(self, class_id, conf, pos):
return [
class_id,
conf,
(pos[1] - (pos[3] / 2)) / self.h,
(pos[0] - (pos[2] / 2)) / self.w,
(pos[1] + (pos[3] / 2)) / self.h,
(pos[0] + (pos[2] / 2)) / self.w,
]
def post_process_yolonas_raw(self, output: list[np.ndarray]):
"""RKNN-style YOLOโ€‘NAS raw postโ€‘process adapted for OpenVINO."""
N = output[0].shape[1]
boxes = output[0].reshape(N, 4)
scores = output[1].reshape(N, self.num_label)
class_ids = np.argmax(scores, axis=1)
scores = scores[np.arange(N), class_ids]
args_best = np.argwhere(scores > self.thresh)[:, 0]
num_matches = len(args_best)
if num_matches == 0:
return np.zeros((20, 6), np.float32)
elif num_matches > 20:
args_best20 = np.argpartition(scores[args_best], -20)[-20:]
args_best = args_best[args_best20]
boxes = boxes[args_best]
class_ids = class_ids[args_best]
scores = scores[args_best]
boxes = np.transpose(
np.vstack(
(
boxes[:, 1] / self.h,
boxes[:, 0] / self.w,
boxes[:, 3] / self.h,
boxes[:, 2] / self.w,
)
)
)
results = np.hstack(
(class_ids[..., np.newaxis], scores[..., np.newaxis], boxes)
)
# Pad to (20, 6)
out = np.zeros((20, 6), np.float32)
out[:results.shape[0], :] = results
return out
def detect_raw(self, tensor_input):
infer_request = self.interpreter.create_infer_request()
expected_type = self.interpreter.inputs[0].get_element_type().to_string()
if expected_type in ("u8", "uint8"):
tensor_input = tensor_input.astype(np.uint8)
else:
tensor_input = tensor_input.astype(np.float32) / 255.0
input_tensor = ov.Tensor(array=tensor_input)
if self.ov_model_type == ModelTypeEnum.dfine:
infer_request.set_tensor("images", input_tensor)
target_sizes_tensor = ov.Tensor(np.array([[self.h, self.w]], dtype=np.int64))
infer_request.set_tensor("orig_target_sizes", target_sizes_tensor)
infer_request.infer()
tensor_output = (
infer_request.get_output_tensor(0).data,
infer_request.get_output_tensor(1).data,
infer_request.get_output_tensor(2).data,
)
return post_process_dfine(tensor_output, self.w, self.h)
infer_request.infer(input_tensor)
detections = np.zeros((20, 6), np.float32)
if self.model_invalid:
return detections
elif self.ov_model_type == ModelTypeEnum.rfdetr:
return post_process_rfdetr([
infer_request.get_output_tensor(0).data,
infer_request.get_output_tensor(1).data,
])
elif self.ov_model_type == ModelTypeEnum.ssd:
results = infer_request.get_output_tensor(0).data[0][0]
for i, (_, class_id, score, xmin, ymin, xmax, ymax) in enumerate(results):
if i == 20:
break
detections[i] = [class_id, float(score), ymin, xmin, ymax, xmax]
return detections
elif self.ov_model_type == ModelTypeEnum.yolonas:
if self.raw_yolonas_mode:
boxes = infer_request.get_output_tensor(0).data
scores = infer_request.get_output_tensor(1).data
return self.post_process_yolonas_raw([boxes, scores])
else:
predictions = infer_request.get_output_tensor(0).data
for i, prediction in enumerate(predictions):
if i == 20:
break
(_, x_min, y_min, x_max, y_max, confidence, class_id) = prediction
if class_id < 0:
break
detections[i] = [
class_id,
confidence,
y_min / self.h,
x_min / self.w,
y_max / self.h,
x_max / self.w,
]
return detections
elif self.ov_model_type == ModelTypeEnum.yologeneric:
out_tensor = [item.data for item in infer_request.output_tensors]
return post_process_yolo(out_tensor, self.w, self.h)
elif self.ov_model_type == ModelTypeEnum.yolox:
out_tensor = infer_request.get_output_tensor()
results = out_tensor.data
results[..., :2] = (results[..., :2] + self.grids) * self.expanded_strides
results[..., 2:4] = np.exp(results[..., 2:4]) * self.expanded_strides
image_pred = results[0, ...]
class_conf = np.max(image_pred[:, 5:5 + self.num_classes], axis=1, keepdims=True)
class_pred = np.argmax(image_pred[:, 5:5 + self.num_classes], axis=1)
class_pred = np.expand_dims(class_pred, axis=1)
conf_mask = (image_pred[:, 4] * class_conf.squeeze() >= 0.3).squeeze()
detections = np.concatenate((image_pred[:, :5], class_conf, class_pred), axis=1)
detections = detections[conf_mask]
ordered = detections[detections[:, 5].argsort()[::-1]][:20]
for i, object_detected in enumerate(ordered):
detections[i] = self.process_yolo(object_detected[6], object_detected[5], object_detected[:4])
return detections
|
Beta Was this translation helpful? Give feedback.
-
|
I'd be interested in hearing more about what code you run to remove postprocessing. But thanks for sharing this, this will be good to keep in mind as NPU support may be worked on more |
Beta Was this translation helpful? Give feedback.
-
|
Great! I have idea from your rknn.py
|
Beta Was this translation helpful? Give feedback.
-
|
I have converted yolonas_s 320x320 model. this model solve "to_shape was called on a dynamic shape" but It's must used with my custom openvino.py. https://github.com/cluangar/patch-frigate-intel-npu/tree/master/Models/yolo_nas_s_320x320 |
Beta Was this translation helpful? Give feedback.
-
|
I am in the process of overhauling my server and wondering if a NPU on a Ultra 245k would be worth it for 8 cameras? |
Beta Was this translation helpful? Give feedback.
-
|
Frigate+ YOLO-NAS can't be run on Intel NPU, but also I haven't seen any results for yolov9 at all |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
I've been trying to follow the (slightly outdated) instructions here to get my Intel Core Ultra i5 245K to work as an NPU detector.
After adjusting things a bit (e.g. 0.16 is based on debian12, so i could simply
apt-get install libtbb12andopenvino==2024.4.0by default), I don't know how exactly to use any Frigate+ model.Using the YOLO-NAS model it generates instead of the default
/openvino-model/ssdlite_mobilenet_v2.xmlyields the following errorVersion
0.16.0-d96efdb
Frigate config file
docker-compose file or Docker CLI command
Relevant Frigate log output
Install method
Proxmox via Docker
Object Detector
OpenVino
Screenshots of the Frigate UI's System metrics pages
N/A
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions