[Detector Support]: HIP error hipErrorInvalidDeviceFunction:invalid device function, how to solve?

[Feratin0s]

Problem Description:

I have an Ubuntu 24.04 LTS server with an AMD RX 5500 XT (gfx1030).

ROCm is installed and rocminfo correctly detects the GPU.

Running Frigate with the stable-rocm image and ONNX models (yolox_tiny.onnx and yolo11n.onnx) results in HIP errors:

hipErrorInvalidDeviceFunction

HIP failure 100: no ROCm-capable device is detected

I tried mapping /dev/kfd and /dev/dri/card1, but the kfd module does not exist in my kernel 6.8.0.

The ONNX models work on CPU but fail on GPU.

I suspect ONNX Runtime does not fully support the RX 5500 XT (gfx1030) with ROCm.

I want to run Frigate with GPU detection, but I'm unsure if it’s possible with my GPU.

What I’ve Tried:

Using HSA_OVERRIDE_GFX_VERSION=10.3.0 and TF_ROCM_USE_IMMEDIATE_MODE=1.

Checking /dev/dri/ and /dev/kfd devices and mapping to the correct GPU.

Testing different ONNX models.

Goal:

Run Frigate with stable ONNX detection.

Version

0.16.0-c2f8de9

Frigate config file

#mqtt:
  host: 192.168.1.1 # Endereço do broker MQTT
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  enabled: true

detectors:
  rocm:
    type: onnx
#    device: GPU
model:
  model_type: yolox
  width: 416 # <--- should match whatever was set in notebook
  height: 416 # <--- should match whatever was set in notebook
  input_dtype: float_denorm
  input_tensor: nchw
  path: /config/models/yolox_tiny.onnx
  labelmap_path: /labelmap/coco-80.txt

cameras:
  camera_quintal: # <------ Name the camera
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://xxxx:[email protected]:554/ # <----- The stream you want to use for detection
          roles:
            - detect
            - record
            #- audio
    detect:
      enabled: true # <---- disable detection until you have a working camera feed
      width: 2304
      height: 2592
      fps: 5
    objects:
      track:
        - person
        - dog
        - cat
    #audio:
    #  enabled: True
    #  listen:
     #   - bark
     #   - scream
     #   - speech
     #   - yell
    record:
      enabled: true
      retain:
        days: 5
      alerts:
        retain:
          days: 10
      detections:
        retain:
          days: 10
    zones:
      Portão:
        coordinates: 0.429,0.12,0.415,0.156,0.488,0.151
        loitering_time: 0

detect:
  enabled: true
version: 0.16-0

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-rocm
    pull_policy: always
    shm_size: "512mb" # update for your cameras based on calculation above
    devices:
       - /dev/kfd:/dev/kfd
       - /dev/dri:/dev/dri
       - /dev/dri/card1:/dev/dri/card1
    group_add:
      - 44 #video
      - 903 #render
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./configs:/config
      - ./models:/config/models
      - ./nvr:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      # - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      #- HSA_OVERRIDE_GFX_VERSION= "10.3.0"
      #- TF_ROCM_USE_IMMEDIATE_MODE=1  # Evita travamentos
      - FRIGATE_RTSP_PASSWORD=xxxxxxxxxx # Senha RTSP
      - PUID=1000
      - PGID=1000

Relevant Frigate log output

s6-rc: info: service nginx successfully started
s6-rc: info: service certsync: starting
s6-rc: info: service certsync successfully started
s6-rc: info: service legacy-services: starting
2025-08-25 23:32:12.234338495  [INFO] Starting certsync...
s6-rc: info: service legacy-services successfully started
2025-08-25 23:32:12.896744683  127.0.0.1 - - [25/Aug/2025:23:32:12 -0300] "" 400 0 "-" "-" "-"
2025-08-25 23:32:14.648418557  [2025-08-25 23:32:14] frigate.util.config            INFO    : Checking if frigate config needs migration...
2025-08-25 23:32:14.669728522  [2025-08-25 23:32:14] frigate.util.config            INFO    : frigate config does not need migration...
2025-08-25 23:32:15.354802667  2025/08/25 23:32:15 [error] 245#245: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", subrequest: "/auth", upstream: "http://127.0.0.1:5001/auth", host: "127.0.0.1:5000"
2025-08-25 23:32:15.354842122  2025/08/25 23:32:15 [error] 245#245: *2 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", host: "127.0.0.1:5000"
2025-08-25 23:32:17.406402170  [2025-08-25 23:32:17] frigate.util.services          INFO    : Automatically detected vaapi hwaccel for video decoding
2025-08-25 23:32:17.471553436  [2025-08-25 23:32:17] frigate.app                    INFO    : Starting Frigate (0.16.0-c2f8de9)
2025-08-25 23:32:17.481731968  [2025-08-25 23:32:17] peewee_migrate.logs            INFO    : Starting migrations
2025-08-25 23:32:17.483984629  [2025-08-25 23:32:17] peewee_migrate.logs            INFO    : There is nothing to migrate
2025-08-25 23:32:17.494964394  [2025-08-25 23:32:17] frigate.app                    INFO    : Recording process started: 702
2025-08-25 23:32:17.501329343  [2025-08-25 23:32:17] frigate.app                    INFO    : Review process started: 704
2025-08-25 23:32:17.505569096  [2025-08-25 23:32:17] frigate.app                    INFO    : go2rtc process pid: 116
2025-08-25 23:32:17.521313135  [2025-08-25 23:32:17] detector.rocm                  INFO    : Starting detection process: 727
2025-08-25 23:32:17.523763003  [2025-08-25 23:32:17] frigate.detectors.plugins.onnx INFO    : ONNX: loaded onnxruntime module
2025-08-25 23:32:17.525098882  [2025-08-25 23:32:17] frigate.detectors.plugins.onnx INFO    : ONNX: loading /config/models/yolox_tiny.onnx
2025-08-25 23:32:17.543705043  [2025-08-25 23:32:17] frigate.app                    INFO    : Output process started: 755
2025-08-25 23:32:17.582240984  [2025-08-25 23:32:17] frigate.app                    INFO    : Camera processor started for camera_quintal: 789
2025-08-25 23:32:17.600753103  [2025-08-25 23:32:17] frigate.app                    INFO    : Capture process started for camera_quintal: 791
2025-08-25 23:32:17.941472819  [2025-08-25 23:32:17] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2025-08-25 23:32:18.283890259  [2025-08-25 23:32:18] frigate.api.fastapi_app        INFO    : FastAPI started
2025-08-25 23:32:20.451925220  [INFO] Starting go2rtc healthcheck service...
2025-08-25 23:32:21.808758886  [2025-08-25 23:32:21] frigate.detectors.plugins.onnx INFO    : ONNX: /config/models/yolox_tiny.onnx loaded
2025-08-25 23:32:21.814444326  2025-08-25 23:32:21.814386393 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running Slice node. Name:'Slice_34' Status Message: HIP error hipErrorInvalidDeviceFunction:invalid device function
2025-08-25 23:32:21.964105799  Process detector:rocm:
2025-08-25 23:32:21.964113924  Traceback (most recent call last):
2025-08-25 23:32:21.964118484    File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2025-08-25 23:32:21.964121575      self.run()
2025-08-25 23:32:21.964125500    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2025-08-25 23:32:21.964130091      return run(*args, **kwargs)
2025-08-25 23:32:21.964166080             ^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:21.964170303    File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
2025-08-25 23:32:21.964175013      self._target(*self._args, **self._kwargs)
2025-08-25 23:32:21.964179046    File "/opt/frigate/frigate/object_detection/base.py", line 136, in run_detector
2025-08-25 23:32:21.964182701      detections = object_detector.detect_raw(input_frame)
2025-08-25 23:32:21.964186272                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:21.964229291    File "/opt/frigate/frigate/object_detection/base.py", line 86, in detect_raw
2025-08-25 23:32:21.964233463      return self.detect_api.detect_raw(tensor_input=tensor_input)
2025-08-25 23:32:21.964238699             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:21.964242631    File "/opt/frigate/frigate/detectors/plugins/onnx.py", line 81, in detect_raw
2025-08-25 23:32:21.964275312      tensor_output = self.model.run(None, {model_input_name: tensor_input})
2025-08-25 23:32:21.964279505                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:21.964284533    File "/usr/local/lib/python3.11/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 266, in run
2025-08-25 23:32:21.964288309      return self._sess.run(output_names, input_feed, run_options)
2025-08-25 23:32:21.964319056             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:21.964325901  onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Slice node. Name:'Slice_34' Status Message: HIP error hipErrorInvalidDeviceFunction:invalid device function
2025-08-25 23:32:37.931950646  [2025-08-25 23:32:37] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2025-08-25 23:32:37.964577293  [2025-08-25 23:32:37] detector.rocm                  INFO    : Starting detection process: 923
2025-08-25 23:32:37.966124701  [2025-08-25 23:32:37] frigate.detectors.plugins.onnx INFO    : ONNX: loaded onnxruntime module
2025-08-25 23:32:37.966523451  [2025-08-25 23:32:37] frigate.detectors.plugins.onnx INFO    : ONNX: loading /config/models/yolox_tiny.onnx
2025-08-25 23:32:42.177721098  [2025-08-25 23:32:42] frigate.detectors.plugins.onnx INFO    : ONNX: /config/models/yolox_tiny.onnx loaded
2025-08-25 23:32:42.184092176  2025-08-25 23:32:42.184020852 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running Slice node. Name:'Slice_34' Status Message: HIP error hipErrorInvalidDeviceFunction:invalid device function
2025-08-25 23:32:42.330141761  Process detector:rocm:
2025-08-25 23:32:42.330147690  Traceback (most recent call last):
2025-08-25 23:32:42.330150809    File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2025-08-25 23:32:42.330152918      self.run()
2025-08-25 23:32:42.330155528    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2025-08-25 23:32:42.330157626      return run(*args, **kwargs)
2025-08-25 23:32:42.330159856             ^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:42.330162473    File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
2025-08-25 23:32:42.330164809      self._target(*self._args, **self._kwargs)
2025-08-25 23:32:42.330167500    File "/opt/frigate/frigate/object_detection/base.py", line 136, in run_detector
2025-08-25 23:32:42.330169865      detections = object_detector.detect_raw(input_frame)
2025-08-25 23:32:42.330172260                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:42.330174799    File "/opt/frigate/frigate/object_detection/base.py", line 86, in detect_raw
2025-08-25 23:32:42.330177222      return self.detect_api.detect_raw(tensor_input=tensor_input)
2025-08-25 23:32:42.330179730             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:42.330182287    File "/opt/frigate/frigate/detectors/plugins/onnx.py", line 81, in detect_raw
2025-08-25 23:32:42.330184858      tensor_output = self.model.run(None, {model_input_name: tensor_input})
2025-08-25 23:32:42.330187452                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:42.330190591    File "/usr/local/lib/python3.11/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 266, in run
2025-08-25 23:32:42.330193034      return self._sess.run(output_names, input_feed, run_options)
2025-08-25 23:32:42.330195462             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-25 23:32:42.330199825  onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Slice node. Name:'Slice_34' Status Message: HIP error hipErrorInvalidDeviceFunction:invalid device function
2025-08-25 23:32:43.009699878  [2025-08-25 23:32:43] frigate.util.services          ERROR   : Unable to poll intel GPU stats: No device filter specified and no discrete/integrated i915 devices found
2025-08-25 23:32:43.009707682
2025-08-25 23:32:47.950182006  [2025-08-25 23:32:47] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service go2rtc-healthcheck: stopping
s6-rc: info: service certsync: stopping

Install method

Docker Compose

Object Detector

Other

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[inkeep-ai-assistant[bot]]

Your error HIP error hipErrorInvalidDeviceFunction:invalid device function with the RX 5500 XT (gfx1030) and ROCm in Frigate indicates a compatibility problem between the ROCm stack, your GPU, and the ONNX runtime. The documentation notes that ROCm can run on AMD discrete GPUs, but support for specific models can depend on driver and kernel compatibility(1)(2). The error about /dev/kfd not being found in your kernel (as shown in your screenshot) means the kernel module for AMD ROCm (kfd) is missing, which is required for ROCm to function(1)(3).

Here are the key troubleshooting and requirements from the documentation:

• ROCm requires the kfd kernel module to be present and loaded for GPU offload. Your kernel (6.8.0) does not have this module available, so ROCm cannot initialize fully(1)(3).
• For ROCm to work, you must have /dev/kfd and /dev/dri mapped into the container(1)(3).
• The documentation explicitly states that "ROCm software stack comes with a limited set of GPU drivers and for newer or missing models you will have to override the chipset version to an older/generic version to get things working." You tried HSA_OVERRIDE_GFX_VERSION=10.3.0, which is the right override for some chipsets, but the kernel module is still required(1)(3).
• To check if ROCm is working, you should run /opt/rocm/bin/rocminfo in the container and verify that both the CPU and GPU are listed. The absence of /dev/kfd will prevent this from succeeding(1)(3).
• If ROCm is not working due to missing kernel support, ONNX models will only run on CPU, not GPU(1)(3).

Solution:
You must install a kernel that includes the ROCm kfd module and ensure that the ROCm drivers support your RX 5500 XT (gfx1030). Without /dev/kfd, ROCm GPU acceleration is not possible in Frigate(1)(3)(4). There is no workaround in Frigate itself for this; it is a system-level requirement. Once the kernel and ROCm stack are correctly installed and /dev/kfd is available, you may need to use the HSA_OVERRIDE_GFX_VERSION environment variable if automatic detection fails(1)(3).

For further ROCm and GPU compatibility information:

• Supported Hardware

📚 Sources:

1. Supported Hardware
2. Supported Hardware
3. Supported Hardware
4. [Detector Support]: ROCm Issues on Ryzen 7 5825u: Data Type Mismatch and HIP Errors with YOLOv5/YOLOv8 Models

---

Have another question? Just tag @inkeep.