[Detector Support]: How do I know if onnx is correctly using TensorRT?

[remz1337]

Describe the problem you are having

The docs mention that the -tensorrt image will automatically detect TensorRT:

Nvidia GPU

ONNX: TensorRT will automatically be detected and used as a detector in the -tensorrt Frigate image when a supported ONNX model is configured.

However, how do I know if it's really using TensorRT execution provider and not only the CUDA execution provider? Is there any way to check? Should I be seeing something in the logs?

Version

0.16-0

Frigate config file

mqtt:
  enabled: false
cameras:
  test:
    ffmpeg:
      inputs:
        - path: /media/frigate/person-bicycle-car-detection.mp4
          input_args: -re -stream_loop -1 -fflags +genpts
          roles:
            - detect
            - rtmp
    detect:
      height: 1080
      width: 1920
      fps: 5
# Optional: Authentication configuration
auth:
  # Optional: Enable authentication
  enabled: false
detect:
  enabled: true
ffmpeg:
  hwaccel_args: preset-nvidia
detectors:
  onnx:
    type: onnx

model:
  model_type: dfine
  width: 320
  height: 320
  input_tensor: nchw
  input_dtype: float
  path: /D-FINE/models/dfine_n_coco.onnx
  labelmap_path: /labelmap/coco-80.txt

version: 0.16.1-0

docker-compose file or Docker CLI command

N/A

Relevant Frigate log output

2025-09-09 23:36:54.674538  [INFO] Preparing Frigate...
2025-09-09 23:36:55.150328  [INFO] Starting Frigate...
2025-09-09 23:36:56.838907  <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
2025-09-09 23:37:00.762798  [2025-09-09 23:37:00] frigate.util.config            INFO    : Checking if frigate config needs migration...
2025-09-09 23:37:00.769370  [2025-09-09 23:37:00] frigate.util.config            INFO    : frigate config does not need migration...
2025-09-09 23:37:00.788538  [2025-09-09 23:37:00] frigate.app                    INFO    : Starting Frigate (0.16.1-)
2025-09-09 23:37:00.798903  [2025-09-09 23:37:00] peewee_migrate.logs            INFO    : Starting migrations
2025-09-09 23:37:00.799177  [2025-09-09 23:37:00] peewee_migrate.logs            INFO    : There is nothing to migrate
2025-09-09 23:37:00.813373  [2025-09-09 23:37:00] frigate.app                    INFO    : Recording process started: 535
2025-09-09 23:37:00.826610  [2025-09-09 23:37:00] frigate.app                    INFO    : Review process started: 548
2025-09-09 23:37:00.828825  [2025-09-09 23:37:00] frigate.app                    INFO    : go2rtc process pid: 162
2025-09-09 23:37:00.847250  [2025-09-09 23:37:00] detector.onnx                  INFO    : Starting detection process: 560
2025-09-09 23:37:00.848053  [2025-09-09 23:37:00] frigate.detectors.plugins.onnx INFO    : ONNX: loaded onnxruntime module
2025-09-09 23:37:00.848356  [2025-09-09 23:37:00] frigate.detectors.plugins.onnx INFO    : ONNX: loading /D-FINE/models/dfine_n_coco.onnx
2025-09-09 23:37:00.853588  �[1;31m2025-09-09 23:37:00.850213758 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 577, index: 1, mask: {2, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m
2025-09-09 23:37:00.853634  �[1;31m2025-09-09 23:37:00.850231698 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 578, index: 2, mask: {3, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m
2025-09-09 23:37:00.853650  �[1;31m2025-09-09 23:37:00.850219248 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 576, index: 0, mask: {0, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m
2025-09-09 23:37:00.881243  �[0;93m2025-09-09 23:37:00.881211750 [W:onnxruntime:, graph.cc:117 MergeShapeInfo] Error merging shape info for output. 'linear_28' source:{1200,1} target:{1200}. Falling back to lenient merge.�[m
2025-09-09 23:37:00.881897  �[0;93m2025-09-09 23:37:00.881884604 [W:onnxruntime:, graph.cc:117 MergeShapeInfo] Error merging shape info for output. 'linear_43' source:{1200,1} target:{1200}. Falling back to lenient merge.�[m
2025-09-09 23:37:00.882273  [2025-09-09 23:37:00] frigate.app                    INFO    : Output process started: 580
2025-09-09 23:37:00.882483  �[0;93m2025-09-09 23:37:00.882469387 [W:onnxruntime:, graph.cc:117 MergeShapeInfo] Error merging shape info for output. 'linear_58' source:{1200,1} target:{1200}. Falling back to lenient merge.�[m
2025-09-09 23:37:00.901619  [2025-09-09 23:37:00] frigate.app                    INFO    : Camera processor started for test: 614
2025-09-09 23:37:00.918027  [2025-09-09 23:37:00] frigate.app                    INFO    : Capture process started for test: 617
2025-09-09 23:37:01.064989  [2025-09-09 23:37:01] frigate.api.fastapi_app        INFO    : Starting FastAPI app
2025-09-09 23:37:01.148772  [2025-09-09 23:37:01] frigate.api.fastapi_app        INFO    : FastAPI started
2025-09-09 23:37:01.627672  [2025-09-09 23:37:01] frigate.detectors.plugins.onnx INFO    : ONNX: /D-FINE/models/dfine_n_coco.onnx loaded
2025-09-09 23:42:53.578969  [2025-09-09 23:42:53] frigate.comms.dispatcher       INFO    : Turning on detection for test

Install method

Proxmox via TTeck Script

Object Detector

Other

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[inkeep-ai-assistant[bot]]

Based on the documentation, when using the -tensorrt Frigate image with ONNX, TensorRT will automatically be detected and used as a detector when a supported ONNX model is configured (1). However, the documentation doesn't specify explicit log messages that confirm TensorRT execution provider usage versus CUDA execution provider.

From your logs, you can see that the ONNX detector is loading successfully:

• ONNX: loaded onnxruntime module (1)
• ONNX: /D-FINE/models/dfine_n_coco.onnx loaded (1)

The fact that you're using the D-FINE model is important to note. According to the documentation, D-FINE models are not supported with ROCm (1), and there's a warning that D-FINE models currently only run on OpenVINO in CPU mode, with GPUs failing to compile the model (2).

For ONNX with Nvidia GPUs, the documentation states that Nvidia GPUs will automatically be detected and used with the ONNX detector in the -tensorrt Frigate image (1). The automatic detection means you shouldn't need additional configuration - if you're using the correct image and have a supported model, TensorRT should be utilized.

Your system metrics showing GPU usage (45.3% detector GPU usage) suggests the GPU is being utilized for inference, which indicates the TensorRT execution provider is likely active rather than falling back to CPU-only execution.

📚 Sources:

1. Supported Hardware
2. Supported Hardware

---

Have another question? Just tag @inkeep.

[NickM-27]

If you run Nvidia-Ami on the host and you see the detector process listed then it is using the GPU

The CUDAExecutionProvider is always used as it is better in many ways for Frigates usage

[remz1337]

Thanks, indeed I see it listed. I was surprised to see a much higher Detector CPU usage with onnx models compared to trt models, so I thought something might be wrong.

[NickM-27]

It can be higher than previously, note that in 0.17 this will be improved #20027

[remz1337]

Awesome, can’t wait to try it out. Thanks!