[HW Accel Support]: NVidia acceleration doesn't work after reboot

[majorcs]

Describe the problem you are having

Hi All,

I have a Frigate installation in a docker environment, using portainer, which has been working for several months without an issue.
It uses an NVIDIA GeForce RTX 3060 card as a HW acceleration backend device. Both the detector and the ffmpeg decoding were working fine, the machine was also rebooted many times, without a problem.

After an unplanned reboot the container can not start any more, as it has a problem with the load of the libcuda.so:

2025-06-24 10:50:09.403018937  [2025-06-24 12:50:09] detector.tensorrt              INFO    : Starting detection process: 441
2025-06-24 10:50:09.407184361  Process detector:tensorrt:
2025-06-24 10:50:09.407190736  Traceback (most recent call last):
2025-06-24 10:50:09.407194740    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2025-06-24 10:50:09.407205840      self.run()
2025-06-24 10:50:09.407211263    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2025-06-24 10:50:09.407214927      return run(*args, **kwargs)
2025-06-24 10:50:09.407259792    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2025-06-24 10:50:09.407263647      self._target(*self._args, **self._kwargs)
2025-06-24 10:50:09.407268329    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2025-06-24 10:50:09.407272053      object_detector = LocalObjectDetector(detector_config=detector_config)
2025-06-24 10:50:09.407275560    File "/opt/frigate/frigate/object_detection.py", line 68, in __init__
2025-06-24 10:50:09.407278874      self.detect_api = create_detector(detector_config)
2025-06-24 10:50:09.407331743    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2025-06-24 10:50:09.407335050      return api(detector_config)
2025-06-24 10:50:09.407373348    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 226, in __init__
2025-06-24 10:50:09.407376668      (cuda_err,) = cuda.cuInit(0)
2025-06-24 10:50:09.407380116    File "cuda/cuda.pyx", line 11325, in cuda.cuda.cuInit
2025-06-24 10:50:09.407383346    File "cuda/ccuda.pyx", line 17, in cuda.ccuda.cuInit
2025-06-24 10:50:09.407386697    File "cuda/_cuda/ccuda.pyx", line 2350, in cuda._cuda.ccuda._cuInit
2025-06-24 10:50:09.407390327    File "cuda/_cuda/ccuda.pyx", line 424, in cuda._cuda.ccuda.cuPythonInit
2025-06-24 10:50:09.407393362  RuntimeError: Failed to dlopen libcuda.so
2025-06-24 10:50:09.407396736  Exception ignored in: <function TensorRtDetector.__del__ at 0x77f500515ca0>
2025-06-24 10:50:09.407433815  Traceback (most recent call last):
2025-06-24 10:50:09.407437453    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 262, in __del__
2025-06-24 10:50:09.407440370      if self.outputs is not None:
2025-06-24 10:50:09.407443644  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'

The docker host has all the NVIDIA dependencies installed, as this container already worked for months. I checked the output of the nvidia-smi, and it works both on the host and within a container:

# sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
Tue Jun 24 10:49:00 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:00:10.0 Off |                  N/A |
|  0%   59C    P8             11W /  170W |       2MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

I'm using the tensorrt version of the docker image:

As I checked the container has the libcuda.so included:

root@b7ef368ab74c:/opt/frigate# find /usr -name '*libcuda*so*'
/usr/local/lib/python3.9/dist-packages/nvidia/cuda_runtime/lib/libcudart.so.12
/usr/local/lib/python3.9/dist-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
/usr/local/cuda/compat/lib.real/libcudadebugger.so.530.30.02
/usr/local/cuda/compat/lib.real/libcuda.so.530.30.02
/usr/local/cuda/compat/lib.real/libcuda.so.1
/usr/local/cuda/compat/lib.real/libcudadebugger.so.1
/usr/local/cuda/compat/lib.real/libcuda.so
/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so
/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12
/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12.1.55
/usr/local/cuda/targets/x86_64-linux/lib/stubs/libcuda.so
root@b7ef368ab74c:/opt/frigate# 

In the config I have the detector enabled and the model referenced which has been built already:

detectors:
  tensorrt:
    type: tensorrt

model:
  path: /config/model_cache/tensorrt/yolov7x-640.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 640
  height: 640

What else should I check to find the problem?

Version

0.15.0-6cb5cfb

Frigate config file

mqtt:
  enabled: true
  host: 192.168.1.3
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  stats_interval: 60

detectors:
  tensorrt:
    type: tensorrt

model:
  path: /config/model_cache/tensorrt/yolov7x-640.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 640
  height: 640

ffmpeg:
  hwaccel_args: preset-nvidia

cameras:
  kamera1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.38:554/
          roles:
            - detect
    objects:
      filters:
        car:
          mask: 0.1,0.17,0.099,0.255,0.141,0.296,0.215,0.255,0.234,0.211,0.222,0.138,0.165,0.142
  kamera2:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.48:554/
          roles:
            - detect
  kapucsengo:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.52:554/
          roles:
            - detect
    objects:
      filters:
        car:
          mask: 
            0.151,0.459,0.147,0.505,0.141,0.576,0.165,0.588,0.2,0.585,0.234,0.57,0.256,0.549,0.244,0.501,0.205,0.47
  tolna:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.13/live.sdp
          roles:
            - detect
  tapo01:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.55:554/stream1
          roles:
            - detect

record:
  enabled: true
  expire_interval: 30
  sync_recordings: false
  retain:
    days: 2
    mode: all
  export:
    timelapse_args: -vf setpts=0.04*PTS -r 30
  alerts:
    retain:
      days: 10
    pre_capture: 5
    post_capture: 5
  detections:
    retain:
      days: 10
    pre_capture: 5
    post_capture: 5
objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
    - bicycle
    - car
    - motorcycle
    - bus
    - cat
    - dog

detect:
  fps: 5

birdseye:
  enabled: true
  restream: false
  width: 1920
  height: 1080
  quality: 8
  mode: continuous
  layout:
    scaling_factor: 2.0
    max_cameras: 4

version: 0.15-1
camera_groups:
  group1:
    order: 1
    icon: LuCamera
    cameras:
      - kamera1
      - kamera2
      - tolna
      - kapucsengo
      - tapo01

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
    networks:
      mynet:
        ipv4_address: 192.168.1.2
    shm_size: "1gb" 
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /nas/docker/frigate/config:/config
      - /nas/docker/frigate/storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    environment:
      FRIGATE_RTSP_PASSWORD: "password"
      YOLO_MODELS: "yolov7x-640"
networks:
    mynet:
        name: mynet
        external: true

Relevant Frigate log output

2025-06-24 10:50:09.403018937  [2025-06-24 12:50:09] detector.tensorrt              INFO    : Starting detection process: 441
2025-06-24 10:50:09.407184361  Process detector:tensorrt:
2025-06-24 10:50:09.407190736  Traceback (most recent call last):
2025-06-24 10:50:09.407194740    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2025-06-24 10:50:09.407205840      self.run()
2025-06-24 10:50:09.407211263    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
2025-06-24 10:50:09.407214927      return run(*args, **kwargs)
2025-06-24 10:50:09.407259792    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2025-06-24 10:50:09.407263647      self._target(*self._args, **self._kwargs)
2025-06-24 10:50:09.407268329    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2025-06-24 10:50:09.407272053      object_detector = LocalObjectDetector(detector_config=detector_config)
2025-06-24 10:50:09.407275560    File "/opt/frigate/frigate/object_detection.py", line 68, in __init__
2025-06-24 10:50:09.407278874      self.detect_api = create_detector(detector_config)
2025-06-24 10:50:09.407331743    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2025-06-24 10:50:09.407335050      return api(detector_config)
2025-06-24 10:50:09.407373348    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 226, in __init__
2025-06-24 10:50:09.407376668      (cuda_err,) = cuda.cuInit(0)
2025-06-24 10:50:09.407380116    File "cuda/cuda.pyx", line 11325, in cuda.cuda.cuInit
2025-06-24 10:50:09.407383346    File "cuda/ccuda.pyx", line 17, in cuda.ccuda.cuInit
2025-06-24 10:50:09.407386697    File "cuda/_cuda/ccuda.pyx", line 2350, in cuda._cuda.ccuda._cuInit
2025-06-24 10:50:09.407390327    File "cuda/_cuda/ccuda.pyx", line 424, in cuda._cuda.ccuda.cuPythonInit
2025-06-24 10:50:09.407393362  RuntimeError: Failed to dlopen libcuda.so
2025-06-24 10:50:09.407396736  Exception ignored in: <function TensorRtDetector.__del__ at 0x77f500515ca0>
2025-06-24 10:50:09.407433815  Traceback (most recent call last):
2025-06-24 10:50:09.407437453    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 262, in __del__
2025-06-24 10:50:09.407440370      if self.outputs is not None:
2025-06-24 10:50:09.407443644  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'

Relevant go2rtc log output

-

FFprobe output from your camera

-

Install method

Docker Compose

Object Detector

TensorRT

Network connection

Wired

Camera make and model

Hikvision

Screenshots of the Frigate UI's System metrics pages

Not feasible, as the system is not running

Any other information that may be helpful

No response

[majorcs]

The docker host OS is Ubuntu 24.04, and it automatically upgraded all the NVIDIA related packages a few weeks ago.
The version of the files went from 550.120-0ubuntu0.24.04.1 to 550.144.03-0ubuntu0.24.04.1, I'm not sure if it could be related to the issue or not...

Start-Date: 2025-05-06  06:07:23
Commandline: /usr/bin/unattended-upgrade
Install: linux-signatures-nvidia-6.8.0-59-generic:amd64 (6.8.0-59.61+1, automatic), linux-modules-nvidia-550-6.8.0-59-generic:amd64 (6.8.0-59.61+1, automatic), linux-modules-6.8.0-59-generic:amd64 (6.8.0-59.61, automatic), linux-image-6.8.0-59-generic:amd64 (6.8.0-59.61, automatic), linux-headers-6.8.0-59-generic:amd64 (6.8.0-59.61, automatic), linux-objects-nvidia-550-6.8.0-59-generic:amd64 (6.8.0-59.61+1, automatic), linux-tools-6.8.0-59:amd64 (6.8.0-59.61, automatic), linux-tools-6.8.0-59-generic:amd64 (6.8.0-59.61, automatic), linux-headers-6.8.0-59:amd64 (6.8.0-59.61, automatic), linux-modules-extra-6.8.0-59-generic:amd64 (6.8.0-59.61, automatic), nvidia-firmware-550-550.144.03:amd64 (550.144.03-0ubuntu0.24.04.1, automatic)
Upgrade: libnvidia-common-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-tools-common:amd64 (6.8.0-58.60, 6.8.0-59.61), libnvidia-fbc1-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), libnvidia-gl-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-headers-generic:amd64 (6.8.0-58.60, 6.8.0-59.61), libnvidia-extra-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), nvidia-compute-utils-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-signatures-nvidia-6.8.0-58-generic:amd64 (6.8.0-58.60, 6.8.0-58.60+1), nvidia-driver-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-modules-nvidia-550-6.8.0-58-generic:amd64 (6.8.0-58.60, 6.8.0-58.60+1), libnvidia-encode-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-modules-nvidia-550-generic:amd64 (6.8.0-58.60, 6.8.0-59.61+1), nvidia-utils-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), xserver-xorg-video-nvidia-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-objects-nvidia-550-6.8.0-58-generic:amd64 (6.8.0-58.60, 6.8.0-58.60+1), libnvidia-decode-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-generic:amd64 (6.8.0-58.60, 6.8.0-59.61), nvidia-kernel-common-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-image-generic:amd64 (6.8.0-58.60, 6.8.0-59.61), libnvidia-cfg1-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), nvidia-kernel-source-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), libnvidia-compute-550:amd64 (550.120-0ubuntu0.24.04.1, 550.144.03-0ubuntu0.24.04.1), linux-libc-dev:amd64 (6.8.0-58.60, 6.8.0-59.61)
End-Date: 2025-05-06  06:10:21

[majorcs]

I have added this to my docker compose, and it solved the issue:

    volumes:
      - /usr/lib/x86_64-linux-gnu/libcuda.so:/usr/lib/x86_64-linux-gnu/libcuda.so
      - /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1
      - /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03:/usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03
      - /usr/lib/x86_64-linux-gnu/libnvcuvid.so:/usr/lib/x86_64-linux-gnu/libnvcuvid.so
      - /usr/lib/x86_64-linux-gnu/libnvcuvid.so.1:/usr/lib/x86_64-linux-gnu/libnvcuvid.so.1
      - /usr/lib/x86_64-linux-gnu/libnvcuvid.so.550.144.03:/usr/lib/x86_64-linux-gnu/libnvcuvid.so.550.144.03
      - /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so:/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so
      - /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1:/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1
      - /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.550.144.03:/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.550.144.03

However I think this is not the correct solution...