[Detector Support]: Developing a detector plugin

[swdee]

Describe the problem you are having

I am wanting to develop a detector plugin for the Orion O6 to make use of its NPU. Before doing any integration with the NPU, my first goal is to purely hook into Frigates detector plugin system.

I have created file detectors/plugins/oriono6.py with contents;

import logging
from typing import Literal

from frigate.detectors.detection_api import DetectionApi
from frigate.detectors.detector_config import BaseDetectorConfig

logger = logging.getLogger(__name__)

DETECTOR_KEY = "oriono6"


class OrionO6DetectorConfig(BaseDetectorConfig):
    type: Literal[DETECTOR_KEY]


class OrionO6(DetectionApi):
    type_key = DETECTOR_KEY

    def __init__(self, config: OrionO6DetectorConfig):
        super().__init__(config)
        self.height = config.model.height
        self.width = config.model.width
        model_path = config.model.path
        self.model_type = config.model.model_type
        logger.info("OrionO6 Init called, config: %s", config.model_dump_json(indent=2))

    # tensor_input is a H×W×C BGR frame?
    def detect_raw(self, tensor_input):
        # raise RuntimeError("🚨 detect_raw() called")
        logger.info(
            "OrionO6 detect_raw called with type=%s, dtype=%s, shape=%s",
            type(tensor_input),
            tensor_input.dtype,
            tensor_input.shape,
        )

Then I connect to the shell of the dev container and start frigate with;

python3 -m frigate

As frigate starts I can see in the log output that the plugin's init() call has been triggered;

[2025-05-01 16:34:03] detector.orion                 INFO    : Starting detection process: 17946
[2025-05-01 16:34:03] frigate.detectors.plugins.oriono6 INFO    : OrionO6 Init called, config: {
  "type": "oriono6",
  "model": {
    "path": "/config/model_cache/oriono6/yolov8_l.cix",
    "labelmap_path": "/labelmap/coco-80.txt",
    "width": 640,
    "height": 640,
    "labelmap": {},
    "attributes_map": {
      "person": [
        "amazon",
        "face"
      ],
      "car": [
        "amazon",
        "an_post",
        "dhl",
        "dpd",
        "fedex",
        "gls",
        "license_plate",
        "nzpost",
        "postnl",
        "postnord",
        "purolator",
        "ups",
        "usps"
      ],
      "motorcycle": [
        "license_plate"
      ]
    },
    "input_tensor": "nhwc",
    "input_pixel_format": "bgr",
    "input_dtype": "int",
    "model_type": "yolo-generic"
  },
  "model_path": null
}
[2025-05-01 16:34:03] frigate.comms.mqtt             DEBUG   : MQTT connected

However I don't see any logging output for calls made to detect_raw(), meaning this function is not being called so detection is not working.

What am I missing with the plugin hooks/configuration?

Version

0.16.0-a9f139e0

Frigate config file

mqtt:
  host: mqtt

logger:
  default: debug
    
detectors:
  orion:
    type: oriono6

model:
  width: 640
  height: 640
  input_tensor: nhwc
  input_pixel_format: bgr
  input_dtype: int
  model_type: yolo-generic
  labelmap_path: /labelmap/coco-80.txt
  path: /config/model_cache/oriono6/yolov8_l.cix

cameras:
  test:
    ffmpeg:
      inputs:
        - path: /media/frigate/palace.mp4
          input_args: -re -stream_loop -1 -fflags +genpts
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720      
      fps: 5
      enabled: true        

version: 0.16-0

docker-compose file or Docker CLI command

Using the default devcontainer.json setup running Frigate in a Dev Container via VS Code.

Relevant Frigate log output

As above.

Install method

Docker Compose

Object Detector

Other

Screenshots of the Frigate UI's System metrics pages

n/a

Any other information that may be helpful

No response

[NickM-27]

Check the logs, you'll likely have an error indicating it couldn't be selected.

Otherwise, are you sure motion has been detected and detect is enabled

[swdee]

I can't see anything in the error log to suggest it was not selected.

vscode ➜ /workspace/frigate (dev) $ python3 -m frigate
/usr/local/lib/python3.11/dist-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/usr/local/lib/python3.11/dist-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/usr/local/lib/python3.11/dist-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/usr/local/lib/python3.11/dist-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
[2025-05-01 16:59:42] frigate.util.config            INFO    : Checking if frigate config needs migration...
[2025-05-01 16:59:42] frigate.util.config            INFO    : frigate config does not need migration...
[2025-05-01 16:59:42] urllib3.connectionpool         DEBUG   : Starting new HTTP connection (1): 127.0.0.1:1984
[2025-05-01 16:59:43] urllib3.connectionpool         DEBUG   : http://127.0.0.1:1984 "GET /api/ffmpeg/hardware HTTP/1.1" 200 266
[2025-05-01 16:59:43] frigate.util.services          WARNING : Did not detect hwaccel, using a GPU for accelerated video decoding is highly recommended
[2025-05-01 16:59:43] frigate.app                    INFO    : Starting Frigate (0.16.0-a9f139e0)
[2025-05-01 16:59:43] frigate.app                    DEBUG   : Skipping directory: /config
[2025-05-01 16:59:43] frigate.app                    DEBUG   : Skipping directory: /media/frigate/recordings
[2025-05-01 16:59:43] frigate.app                    DEBUG   : Skipping directory: /media/frigate/clips/thumbs
[2025-05-01 16:59:43] frigate.app                    DEBUG   : Skipping directory: /media/frigate/clips/cache
[2025-05-01 16:59:43] frigate.app                    DEBUG   : Skipping directory: /tmp/cache
[2025-05-01 16:59:43] frigate.app                    DEBUG   : Skipping directory: /config/model_cache
[2025-05-01 16:59:43] frigate.app                    DEBUG   : Skipping directory: /media/frigate/exports
[2025-05-01 16:59:43] peewee                         DEBUG   : ('CREATE TABLE IF NOT EXISTS "migratehistory" ("id" INTEGER NOT NULL PRIMARY KEY, "name" VARCHAR(255) NOT NULL, "migrated_at" DATETIME NOT NULL)', [])
[2025-05-01 16:59:43] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."name", "t1"."migrated_at" FROM "migratehistory" AS "t1" ORDER BY "t1"."id"', [])
[2025-05-01 16:59:43] peewee_migrate.logs            INFO    : Starting migrations
[2025-05-01 16:59:43] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."name", "t1"."migrated_at" FROM "migratehistory" AS "t1" ORDER BY "t1"."id"', [])
[2025-05-01 16:59:43] peewee_migrate.logs            INFO    : There is nothing to migrate
[2025-05-01 16:59:43] frigate.app                    INFO    : Recording process started: 2910
[2025-05-01 16:59:43] frigate.app                    INFO    : Review process started: 2914
[2025-05-01 16:59:43] frigate.app                    INFO    : go2rtc process pid: 124
[2025-05-01 16:59:43] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 16:59:43] detector.orion                 INFO    : Starting detection process: 2933
[2025-05-01 16:59:43] frigate.detectors.plugins.oriono6 INFO    : OrionO6 Init called, config: {
  "type": "oriono6",
  "model": {
    "path": "/config/model_cache/oriono6/yolov8_l.cix",
    "labelmap_path": "/labelmap/coco-80.txt",
    "width": 640,
    "height": 640,
    "labelmap": {},
    "attributes_map": {
      "person": [
        "amazon",
        "face"
      ],
      "car": [
        "amazon",
        "an_post",
        "dhl",
        "dpd",
        "fedex",
        "gls",
        "license_plate",
        "nzpost",
        "postnl",
        "postnord",
        "purolator",
        "ups",
        "usps"
      ],
      "motorcycle": [
        "license_plate"
      ]
    },
    "input_tensor": "nhwc",
    "input_pixel_format": "bgr",
    "input_dtype": "int",
    "model_type": "yolo-generic"
  },
  "model_path": null
}
[2025-05-01 16:59:43] frigate.app                    INFO    : Output process started: 2949
[2025-05-01 16:59:43] frigate.comms.mqtt             DEBUG   : MQTT connected
[2025-05-01 16:59:43] peewee.sqliteq                 DEBUG   : received query DELETE FROM "regions" WHERE NOT ("regions"."camera" IN (?))
[2025-05-01 16:59:43] peewee                         DEBUG   : ('DELETE FROM "regions" WHERE NOT ("regions"."camera" IN (?))', ['test'])
[2025-05-01 16:59:43] peewee                         DEBUG   : ('SELECT "t1"."camera", "t1"."grid", "t1"."last_update" FROM "regions" AS "t1" WHERE ("t1"."camera" = ?) LIMIT ? OFFSET ?', ['test', 1, 0])
[2025-05-01 16:59:43] peewee                         DEBUG   : ('SELECT "t1"."id" FROM "event" AS "t1" WHERE ((("t1"."camera" = ?) AND (("t1"."false_positive" IS NULL) OR ("t1"."false_positive" = ?))) AND ("t1"."start_time" > ?))', ['test', False, 0])
[2025-05-01 16:59:43] frigate.util.object            DEBUG   : Found 0 new events for test
[2025-05-01 16:59:43] frigate.app                    INFO    : Camera processor started for test: 2977
[2025-05-01 16:59:43] frigate.app                    DEBUG   : Calculated total camera size 206.0 / 1.6 :: 50 frames for each camera in SHM
[2025-05-01 16:59:43] frigate.app                    INFO    : Capture process started for test: 2984
[2025-05-01 16:59:43] peewee                         DEBUG   : ('SELECT COUNT(?) FROM "recordings" AS "t1" WHERE (("t1"."camera" = ?) AND ("t1"."segment_size" > ?))', ['*', 'test', 0.0])
[2025-05-01 16:59:43] urllib3.connectionpool         DEBUG   : Starting new HTTPS connection (1): api.github.com:443
[2025-05-01 16:59:43] peewee                         DEBUG   : ('SELECT AVG("t1"."segment_size" / ("t1"."end_time" - "t1"."start_time")) FROM "recordings" AS "t1" WHERE (("t1"."camera" = ?) AND ("t1"."segment_size" > ?)) LIMIT ?', ['test', 0.0, 100])
[2025-05-01 16:59:43] frigate.storage                DEBUG   : test has a bandwidth of 0 MiB/hr.
[2025-05-01 16:59:43] frigate.video                  DEBUG   : Starting all ffmpeg processes for test
[2025-05-01 16:59:43] tzlocal                        DEBUG   : /etc/timezone found, contents:
 Etc/UTC

[2025-05-01 16:59:43] tzlocal                        DEBUG   : /etc/localtime found
[2025-05-01 16:59:43] tzlocal                        DEBUG   : 2 found:
 {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
[2025-05-01 16:59:43] peewee.sqliteq                 DEBUG   : received query UPDATE "reviewsegment" SET "end_time" = ? WHERE ("reviewsegment"."end_time" IS NULL)
[2025-05-01 16:59:43] peewee                         DEBUG   : ('UPDATE "reviewsegment" SET "end_time" = ? WHERE ("reviewsegment"."end_time" IS NULL)', [1746089983.394338])
[2025-05-01 16:59:43] urllib3.connectionpool         DEBUG   : https://api.github.com:443 "GET /repos/blakeblackshear/frigate/releases/latest HTTP/1.1" 200 1020
[2025-05-01 16:59:43] peewee.sqliteq                 DEBUG   : received query UPDATE "event" SET "end_time" = ("event"."start_time" + ?) WHERE ("event"."end_time" IS NULL)
[2025-05-01 16:59:43] peewee                         DEBUG   : ('UPDATE "event" SET "end_time" = ("event"."start_time" + ?) WHERE ("event"."end_time" IS NULL)', [30])
[2025-05-01 16:59:43] peewee                         DEBUG   : ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1") AS "_wrapped"', [])
[2025-05-01 16:59:43] frigate.api.fastapi_app        INFO    : Starting FastAPI app
[2025-05-01 16:59:43] frigate.api.auth               DEBUG   : Using jwt secret from .jwt_secret file in config directory.
[2025-05-01 16:59:43] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 16:59:43] frigate.api.fastapi_app        INFO    : FastAPI started
[2025-05-01 16:59:48] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 16:59:53] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 16:59:58] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:00] frigate.output.preview         DEBUG   : successfully saved preview
[2025-05-01 17:00:00] peewee.sqliteq                 DEBUG   : received query INSERT INTO "previews" ("id", "camera", "path", "start_time", "end_time", "duration") VALUES (?, ?, ?, ?, ?, ?)
[2025-05-01 17:00:00] peewee                         DEBUG   : ('INSERT INTO "previews" ("id", "camera", "path", "start_time", "end_time", "duration") VALUES (?, ?, ?, ?, ?, ?)', ['test_1746090000.086178', 'test', '/media/frigate/clips/previews/test/1746086400.125912-1746090000.086178.mp4', 1746086400.125912, 1746090000.086178, 3599.9602661132812])
[2025-05-01 17:00:03] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:08] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:08] frigate.stats.emitter          DEBUG   : Starting stats collection
[2025-05-01 17:00:08] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:08] frigate.stats.emitter          DEBUG   : Finished stats collection
[2025-05-01 17:00:13] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:18] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:23] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:23] frigate.stats.emitter          DEBUG   : Starting stats collection
[2025-05-01 17:00:23] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:23] frigate.stats.emitter          DEBUG   : Finished stats collection
[2025-05-01 17:00:23] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
[2025-05-01 17:00:25] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
[2025-05-01 17:00:26] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
[2025-05-01 17:00:26] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
[2025-05-01 17:00:26] urllib3.connectionpool         DEBUG   : Starting new HTTP connection (1): 127.0.0.1:1984
[2025-05-01 17:00:26] urllib3.connectionpool         DEBUG   : http://127.0.0.1:1984 "GET /api/streams HTTP/1.1" 200 3
[2025-05-01 17:00:26] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."camera", "t1"."start_time", "t1"."end_time", "t1"."severity", "t1"."thumb_path", "t1"."data", COALESCE("t2"."has_been_reviewed", ?) AS "has_been_reviewed" FROM "reviewsegment" AS "t1" LEFT OUTER JOIN "userreviewstatus" AS "t2" ON (("t1"."id" = "t2"."review_segment_id") AND ("t2"."user_id" = ?)) WHERE ((((("t1"."start_time" > ?) AND ("t1"."start_time" < ?)) AND (("t1"."end_time" IS NULL) OR ("t1"."end_time" < ?))) AND ("t1"."severity" = ?)) AND (("t2"."has_been_reviewed" = ?) OR ("t2"."has_been_reviewed" IS NULL))) ORDER BY "t1"."start_time" DESC LIMIT ?', [False, 'anonymous', 1746003626.271038, 1746090026.271025, 1746090026.271025, <SeverityEnum.alert: 'alert'>, False, 10])
[2025-05-01 17:00:28] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:28] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
[2025-05-01 17:00:28] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
[2025-05-01 17:00:33] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:38] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:38] frigate.stats.emitter          DEBUG   : Starting stats collection
[2025-05-01 17:00:38] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:38] frigate.stats.emitter          DEBUG   : Finished stats collection
[2025-05-01 17:00:43] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : Start expire recordings.
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : Start deleted cameras.
[2025-05-01 17:00:43] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."path" FROM "recordings" AS "t1" WHERE (("t1"."camera" NOT IN (?)) AND ("t1"."end_time" < ?))', ['test', 1746090043.536609])
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : Expiring 0 recordings
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : End deleted cameras.
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : Start all cameras.
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : Start camera: test.
[2025-05-01 17:00:43] peewee                         DEBUG   : ('SELECT "t1"."id" FROM "reviewsegment" AS "t1" WHERE (("t1"."camera" = ?) AND ((("t1"."severity" = ?) AND ("t1"."end_time" < ?)) OR (("t1"."severity" = ?) AND ("t1"."end_time" < ?))))', ['test', 'alert', 1745226043.540091, 'detection', 1745226043.540091])
[2025-05-01 17:00:43] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."start_time", "t1"."end_time", "t1"."path", "t1"."objects", "t1"."motion" FROM "recordings" AS "t1" WHERE (("t1"."camera" = ?) AND ("t1"."end_time" < ?)) ORDER BY "t1"."start_time"', ['test', 1746090043.540091])
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : Expiring 0 recordings
[2025-05-01 17:00:43] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."start_time", "t1"."end_time", "t1"."path" FROM "previews" AS "t1" WHERE (("t1"."camera" = ?) AND ("t1"."end_time" < ?)) ORDER BY "t1"."start_time"', ['test', 1746090043.540091])
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : Expiring 1 previews
[2025-05-01 17:00:43] peewee.sqliteq                 DEBUG   : received query DELETE FROM "previews" WHERE ("previews"."id" IN (?))
[2025-05-01 17:00:43] peewee                         DEBUG   : ('DELETE FROM "previews" WHERE ("previews"."id" IN (?))', ['test_1746090000.086178'])
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : End camera: test.
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : End all cameras.
[2025-05-01 17:00:43] frigate.record.cleanup         DEBUG   : End expire recordings.
[2025-05-01 17:00:48] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:53] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:53] frigate.stats.emitter          DEBUG   : Starting stats collection
[2025-05-01 17:00:53] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:00:53] frigate.stats.emitter          DEBUG   : Finished stats collection
[2025-05-01 17:00:58] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:01:03] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:01:08] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:01:08] frigate.stats.emitter          DEBUG   : Starting stats collection
[2025-05-01 17:01:08] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:01:08] frigate.stats.emitter          DEBUG   : Finished stats collection
[2025-05-01 17:01:10] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
[2025-05-01 17:01:10] frigate.comms.ws               DEBUG   : Publishing mqtt message from websockets at onConnect.
[2025-05-01 17:01:10] urllib3.connectionpool         DEBUG   : Starting new HTTP connection (1): 127.0.0.1:1984
[2025-05-01 17:01:10] urllib3.connectionpool         DEBUG   : http://127.0.0.1:1984 "GET /api/streams HTTP/1.1" 200 3
[2025-05-01 17:01:10] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."camera", "t1"."start_time", "t1"."end_time", "t1"."severity", "t1"."thumb_path", "t1"."data", COALESCE("t2"."has_been_reviewed", ?) AS "has_been_reviewed" FROM "reviewsegment" AS "t1" LEFT OUTER JOIN "userreviewstatus" AS "t2" ON (("t1"."id" = "t2"."review_segment_id") AND ("t2"."user_id" = ?)) WHERE ((((("t1"."start_time" > ?) AND ("t1"."start_time" < ?)) AND (("t1"."end_time" IS NULL) OR ("t1"."end_time" < ?))) AND ("t1"."severity" = ?)) AND (("t2"."has_been_reviewed" = ?) OR ("t2"."has_been_reviewed" IS NULL))) ORDER BY "t1"."start_time" DESC LIMIT ?', [False, 'anonymous', 1746003670.748692, 1746090070.748682, 1746090070.748682, <SeverityEnum.alert: 'alert'>, False, 10])
[2025-05-01 17:01:13] asyncio                        DEBUG   : Using selector: EpollSelector
[2025-05-01 17:01:18] asyncio                        DEBUG   : Using selector: EpollSelector

Detect is confirmed enabled.

And debug shows motion is detected.

Any other ideas?

[NickM-27]

you might have too much motion and the motion detection isn't able to calibrate. If you enable regions and see green boxes then that means detection should be running

[swdee]

That was it thank you! I swapped to a different MP4 video and now it logs as anticipated.

[2025-05-02 04:22:25] frigate.detectors.plugins.oriono6 INFO    : OrionO6 detect_raw called with type=<class 'numpy.ndarray'>, dtype=uint8, shape=(1, 640, 640, 3)

[NickM-27]

Feel free to create a new discussion if you have any other questions that come up