[Detector Support]: ONNX GeForce GTX 750 Ti

[nordeep]

Describe the problem you are having

Another request regarding Frigate 0.16 compatibility with GPU. My GTX 750 Ti (Driver Version: 560.35.05, CUDA Version: 12.6) is not working with the ONNX detector and YoloNAS model on Frigate 0.16

According to the CUDA GPU Compute Capability - the GeForce GTX 750 Ti has Compute Capability 5.0, which meets the minimum requirements

I must say that the GTX 750 Ti is compatible with Frigate 0.15 and works fully in my environment.

Version

0.16.1-e664cb2

Frigate config file

mqtt:
  enabled: false

detectors:
  onnx:
    type: onnx
    device: '0'
logger:
  default: info

go2rtc:
  streams:
    eye-camera:
      - rtsp://xxxxxx
  webrtc:
    candidates:
      - 192.168.xx.xx:8555
      - stun:8555

model:
  path: /config/model_cache/tensorrt/yolo_nas_s.onnx
  model_type: yolonas
  width: 320 # <--- should match whatever was set in notebook
  height: 320 # <--- should match whatever was set in notebook
  input_pixel_format: bgr
  input_tensor: nchw
  labelmap_path: /labelmap/coco-80.txt

record:
  enabled: true
  alerts:
    retain:
      days: 4
  detections:
    retain:
      days: 4
cameras:
  eye-camera:
    enabled: true
    mqtt:
      timestamp: false
      bounding_box: false
      crop: false
      quality: 100
      height: 1080
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/eye-camera
          input_args: preset-rtsp-restream-low-latency
          roles:
            - detect
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    objects:
      track:
        - person
    snapshots:
      required_zones:
        - door_zone
      enabled: true
    motion:
      mask:
        - 0,720,613,720,614,0,0,0
        - 1280,38,1280,68,967,69,971,42
    zones:
      door_zone:
        coordinates: 864,212,843,720,598,720,604,208
detect:
  enabled: true

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate-16
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.16.1-tensorrt
    shm_size: "96mb" # update for your cameras based on calculation above
    deploy:    # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
              #device_ids: ['0'] # this is only needed when using multiple GPUs
              count: all # number of GPUs
              capabilities: [gpu]
    cap_add:
      - CAP_PERFMON
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILIIES=all
      - FRIGATE_RTSP_PASSWORD="xxxx"
      - USE_FP16=false
    devices:
      - /dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/config:/config
      - /mnt/frigate/media:/media/frigate
      - /opt/frigate/trt-models:/trt-models
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5001:5000"
      - "1984:1984" # go2rtc
      - "1935:1935" # RTMP feeds
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

Relevant Frigate log output

[E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running Cast node. Name:'/pre_process/pre_process.0/Cast' Status Message: CUDA error cudaErrorNoKernelImageForDevice:no kernel image is available for execution on the device

Install method

Docker Compose

Object Detector

TensorRT

Screenshots of the Frigate UI's System metrics pages

Can't start

Any other information that may be helpful

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05              Driver Version: 560.35.05      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 750 Ti      On  |   00000000:01:00.0 Off |                  N/A |
| 42%   40C    P8              1W /   38W |       2MiB /   2048MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[inkeep-ai-assistant[bot]]

Your GeForce GTX 750 Ti has Compute Capability 5.0, which meets the minimum compute capability required for Frigate’s TensorRT/ONNX support(1). However, the documentation specifies that the minimum driver version must be >=545 and the GPU must support CUDA 12.x libraries for TensorRT(1). The GTX 750 Ti is a Maxwell-era GPU, but not all Maxwell GPUs are fully supported by TensorRT with the CUDA 12.x stack, and some features/models (like YOLO-NAS) may require newer GPU instructions not present on older cards(1).

The error message:

Non-zero status code returned while running Cast node. Name:'/pre_process/pre_process.0/Cast' Status Message: CUDA error cudaErrorNoKernelImageForDevice:no kernel image is available for execution on the device

indicates that the CUDA runtime could not find a compatible kernel for your device. This typically means that the ONNX runtime (or the model itself) was built targeting newer architectures and is not compatible with the GTX 750 Ti(2).

Key points from the documentation:

• Frigate can use ONNX/TensorRT for Nvidia GPUs with Compute Capability 5.0 or greater and CUDA 12.x(1).
• Not all ONNX models are guaranteed to work on all GPUs, especially older ones(2).
• The YOLO-NAS model is supported in ONNX, but hardware compatibility is not guaranteed for legacy cards(2).
• Inference speeds and compatibility are documented for newer cards (e.g., GTX 1070 and above), but there is no guarantee for 750 Ti(1).

Summary:
The GTX 750 Ti meets the minimum compute capability, but the CUDA error suggests either a model incompatibility or lack of support for the necessary CUDA kernels in your environment. This is a known limitation with legacy GPUs and modern ONNX/TensorRT pipelines(2)(1). For best compatibility, consider upgrading to a newer Nvidia GPU (Pascal/Volta/Turing/Ampere or later).

📚 Sources:

1. Recommended hardware
2. Supported Hardware

---

Have another question? Just tag @inkeep.

[nordeep]

@inkeep I performed a number of additional checks.
I installed onnxruntime-gpu
pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
and ran a small test that successfully initialized the CUDA provider in my environment.

import onnxruntime as ort

# 
providers = ort.get_available_providers()
print("Available Providers:", providers)

if 'CUDAExecutionProvider' in providers:
    print("CUDA available")
    
    # 
    try:
        # 
        ort.capi._pybind_state.get_available_providers()
        print("CUDA successfully initialized")
    except Exception as e:
        print(f"CUDA initialization error: {e}")
else:
    print("CUDA NOT available")

[NickM-27]

The version of ONNX runtime used seems to not support certain kernels which causes this issue. I'm still trying to understand exactly which cards are no longer supported, but it seems this is likely due to using the newer onnx runtime version

[nordeep]

The version of ONNX runtime used seems to not support certain kernels which causes this issue. I'm still trying to understand exactly which cards are no longer supported, but it seems this is likely due to using the newer onnx runtime version

From what I've managed to find out, the latest version of TensorFlow that supports Compute Capability 5.0 is 2.9.0.

However, according to the Tensorflow blog post - the latest version with Maxwell support(Compute Capability 5.0) is 2.16.

[nordeep]

@NickM-27, could you try building a Docker image with tensorflow==2.16?
I tried to do it myself, but your Dockerfile uses a devcontainer which is not available to me.

[NickM-27]

The dev container is only for development, the docker builds just build normally

Also, tensorflow is not relevant as tensorflow is not a dependency in 0.16