HailoRT fails with HAILO_UNSUPPORTED_FW_VERSION on Frigate startup (FW 4.21.0 not supported)

[farissi]

Describe the problem you are having

I'm trying to use the Hailo-8 M.2 PCIe accelerator with Frigate and the HailoRT plugin. The Frigate container is running inside a VM with Ubuntu 24.10 on Proxmox, and the HailoRT PCIe driver version is 4.21.0.

At startup, Frigate fails to initialize the Hailo-8 detector with the following error:

[HailoRT] [warning] Unsupported firmware operation. Host: 4.19.0, Device: 4.21.0  
[HailoRT] [error] Operation 51 is not allowed when FW version is not supported.  
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_FW_VERSION(16)  
frigate.detectors.plugins.hailo8l ERROR: HailoRTException during initialization: libhailort failed with error: 16 (HAILO_UNSUPPORTED_FW_VERSION)

It seems the HailoRT version used in Frigate only supports FW 4.19.0, while my device runs firmware 4.21.0.

Important: I do not want to downgrade the firmware.
Is there a way to upgrade the Docker version (or Frigate container) to match the newer firmware, or is it too complex?
I'm worried about losing features or fixes by downgrading my firmware.

Environment:

• Frigate Version: 0.15.0-6cb5cfb
• HailoRT PCIe Driver: 4.21.0
• Device: Hailo-8 M.2 (Yahboom PCIe)
• Host: Ubuntu 24.10 (Kernel 6.11.0-21)
• Running in: Proxmox VM

Thanks in advance!

Version

0.15.1-h8l

Frigate config file

mqtt:
  host: XXX.XXX.X.XX  # change to your MQTT broker

detectors:
  hailo8l:
    type: hailo8l
    device: PCIe

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  model_type: ssd
  path: /config/model_cache/h8l_cache/ssd_mobilenet_v1.hef

logger:
  default: info
  logs:
    frigate.detectors.plugins.hailo8l: debug

cameras:
  cars:
    ffmpeg:
      inputs:
        - path: file:/media/frigate/cars.mp4
          input_args: -stream_loop -1 -re
          roles:
            - detect
    detect:
      enabled: true
      width: 640
      height: 360
      fps: 5
    objects:
      track:
        - person
        - car
version: 0.15-1

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    privileged: true
    image: ghcr.io/blakeblackshear/frigate:0.15.1-h8l
    shm_size: "512mb"
    devices:
      - /dev/hailo0:/dev/hailo0
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - /home/dai/media/frigate:/media/frigate
    ports:
      - "5000:5000"
      - "8554:8554"
      - "8555:8555"
      - "1984:1984"
    environment:
      - FRIGATE_RTSP_PASSWORD=fsracma

Relevant Frigate log output

2025-04-19 00:22:49.624664259  [INFO] Starting go2rtc...
2025-04-19 00:22:49.633413816  127.0.0.1 - - [19/Apr/2025:00:22:49 +0000] "" 400 0 "-" "-" "-"
2025-04-19 00:22:49.690041639  00:22:49.689 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
2025-04-19 00:22:49.690050667  00:22:49.689 INF config path=/dev/shm/go2rtc.yaml
2025-04-19 00:22:49.690209364  00:22:49.690 INF [rtsp] listen addr=:8554
2025-04-19 00:22:49.690292630  00:22:49.690 INF [webrtc] listen addr=:8555/tcp
2025-04-19 00:22:49.690423231  00:22:49.690 INF [api] listen addr=:1984
2025-04-19 00:22:52.200378731  [2025-04-19 00:22:52] frigate.util.config            INFO    : Checking if frigate config needs migration...
2025-04-19 00:22:52.200419281  [2025-04-19 00:22:52] frigate.util.config            INFO    : frigate config does not need migration...
2025-04-19 00:22:52.226202984  [2025-04-19 00:22:52] frigate.util.services          WARNING : Did not detect hwaccel, using a GPU for accelerated video decoding is highly recommended
2025-04-19 00:22:52.252283948  [2025-04-19 00:22:52] frigate.app                    INFO    : Starting Frigate (0.15.0-6cb5cfb)
2025-04-19 00:22:52.262579433  [2025-04-19 00:22:52] peewee_migrate.logs            INFO    : Starting migrations
2025-04-19 00:22:52.262785494  [2025-04-19 00:22:52] peewee_migrate.logs            INFO    : There is nothing to migrate
2025-04-19 00:22:52.262806064  [2025-04-19 00:22:52] frigate.app                    INFO    : Recording process started: 363
2025-04-19 00:22:52.262806776  [2025-04-19 00:22:52] frigate.app                    INFO    : Review process started: 365
2025-04-19 00:22:52.264127293  [2025-04-19 00:22:52] frigate.app                    INFO    : go2rtc process pid: 98
2025-04-19 00:22:52.273178423  [2025-04-19 00:22:52] detector.hailo8l               INFO    : Starting detection process: 379
2025-04-19 00:22:52.273258572  [2025-04-19 00:22:52] frigate.detectors.plugins.hailo8l INFO    : Initializing Hailo device as PCIe
2025-04-19 00:22:52.279737735  [2025-04-19 00:22:52] frigate.app                    INFO    : Output process started: 395
2025-04-19 00:22:52.289919854  [2025-04-19 00:22:52] frigate.app                    INFO    : Camera processor started for cars: 410
2025-04-19 00:22:52.296871476  [2025-04-19 00:22:52] frigate.app                    INFO    : Camera processor started for cars2: 418
2025-04-19 00:22:52.303811435  [2025-04-19 00:22:52] frigate.app                    INFO    : Camera processor started for people: 425
2025-04-19 00:22:52.309284003  [2025-04-19 00:22:52] frigate.app                    INFO    : Camera processor started for people2: 439
2025-04-19 00:22:52.316028022  [2025-04-19 00:22:52] frigate.app                    INFO    : Capture process started for cars: 449
2025-04-19 00:22:52.350490945  [2025-04-19 00:22:52] frigate.app                    INFO    : Capture process started for cars2: 475
2025-04-19 00:22:52.357168762  [2025-04-19 00:22:52] frigate.app                    INFO    : Capture process started for people: 490
2025-04-19 00:22:52.381939248  [2025-04-19 00:22:52] frigate.app                    INFO    : Capture process started for people2: 506
2025-04-19 00:22:52.389328592  [2025-04-19 00:22:52] frigate.detectors.plugins.hailo8l INFO    : A model file already exists at /config/model_cache/h8l_cache/ssd_mobilenet_v1.hef not downloading one.
2025-04-19 00:22:52.450723221  [HailoRT] [warning] Unsupported firmware operation. Host: 4.19.0, Device: 4.21.0
2025-04-19 00:22:52.472667203  [HailoRT] [error] Operation 51 is not allowed when FW version in not supported. Host supported FW version is 4.19.0
2025-04-19 00:22:52.472770880  [HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_FW_VERSION(16)
2025-04-19 00:22:52.472786130  [HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_FW_VERSION(16)
2025-04-19 00:22:52.472799356  [HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_FW_VERSION(16) - Can't get device extended info
2025-04-19 00:22:52.472812131  [HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_FW_VERSION(16)
2025-04-19 00:22:52.472824597  [HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_FW_VERSION(16)
2025-04-19 00:22:52.473854245  [2025-04-19 00:22:52] frigate.detectors.plugins.hailo8l ERROR   : HailoRTException during initialization: libhailort failed with error: 16 (HAILO_UNSUPPORTED_FW_VERSION)
2025-04-19 00:22:52.539910670  Process detector:hailo8l:
2025-04-19 00:22:52.539920009  Traceback (most recent call last):
2025-04-19 00:22:52.539923205    File "/usr/local/lib/python3.9/dist-packages/hailo_platform/pyhailort/pyhailort.py", line 3531, in configure
2025-04-19 00:22:52.539925509      configured_ngs_handles = self._vdevice.configure(hef._hef, configure_params_by_name)
2025-04-19 00:22:52.539935720  hailo_platform.pyhailort._pyhailort.HailoRTStatusException: 16
2025-04-19 00:22:52.539958876  
2025-04-19 00:22:52.539960600  The above exception was the direct cause of the following exception:
2025-04-19 00:22:52.539962924  
2025-04-19 00:22:52.539964196  Traceback (most recent call last):
2025-04-19 00:22:52.539966040    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2025-04-19 00:22:52.539967283      self.run()
2025-04-19 00:22:52.539968696    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2025-04-19 00:22:52.539994927      return run(*args, **kwargs)
2025-04-19 00:22:52.539996481    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2025-04-19 00:22:52.539998044      self._target(*self._args, **self._kwargs)
2025-04-19 00:22:52.539999567    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2025-04-19 00:22:52.540010839      object_detector = LocalObjectDetector(detector_config=detector_config)
2025-04-19 00:22:52.540012422    File "/opt/frigate/frigate/object_detection.py", line 68, in __init__
2025-04-19 00:22:52.540013985      self.detect_api = create_detector(detector_config)
2025-04-19 00:22:52.540015519    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2025-04-19 00:22:52.540016701      return api(detector_config)
2025-04-19 00:22:52.540018555    File "/opt/frigate/frigate/detectors/plugins/hailo8l.py", line 80, in __init__
2025-04-19 00:22:52.540020038      self.network_groups = self.target.configure(self.hef, self.configure_params)
2025-04-19 00:22:52.540021560    File "/usr/local/lib/python3.9/dist-packages/hailo_platform/pyhailort/pyhailort.py", line 3531, in configure
2025-04-19 00:22:52.540031961      configured_ngs_handles = self._vdevice.configure(hef._hef, configure_params_by_name)
2025-04-19 00:22:52.540033694    File "/usr/local/lib/python3.9/dist-packages/hailo_platform/pyhailort/pyhailort.py", line 111, in __exit__
2025-04-19 00:22:52.540035058      self._raise_indicative_status_exception(value)
2025-04-19 00:22:52.540047763    File "/usr/local/lib/python3.9/dist-packages/hailo_platform/pyhailort/pyhailort.py", line 156, in _raise_indicative_status_exception
2025-04-19 00:22:52.540049556      raise self.create_exception_from_status(error_code) from libhailort_exception
2025-04-19 00:22:52.540051189  hailo_platform.pyhailort.pyhailort.HailoRTException: libhailort failed with error: 16 (HAILO_UNSUPPORTED_FW_VERSION)
2025-04-19 00:22:52.559541290  [2025-04-19 00:22:52] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2025-04-19 00:22:52.626566824  [2025-04-19 00:22:52] frigate.api.fastapi_app        INFO    : FastAPI started
2025-04-19 00:22:59.375055605  [INFO] Starting go2rtc healthcheck service...
2025-04-19 00:23:12.552693983  [2025-04-19 00:23:12] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...

Install method

Docker Compose

Object Detector

hailo8l

Network connection

Wired

[NickM-27]

The only way is that downgrade the Hailo driver. There are very few changes between versions so nothing will really be lost.

Ideally in the future the Hailo driver won't have to exactly match versions so this won't be a problem

[farissi]

Thanks for the clarification, that makes sense.

I was hoping to avoid downgrading, but given that the functional differences between versions are minimal, I understand it's currently the most practical path. I’ll proceed with aligning the driver version accordingly.

And agreed, having a more flexible version compatibility in the future would definitely improve the developer experience and reduce these minor roadblocks during setup.

Appreciate your insight!