CUDA driver version is insuffient for CUDA runtime version.

[JayCroghan]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

I updated my Nvidia drivers today and now my Immich ML console log is filled with the following error:

2025-10-12 17:48:07.182877091 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 35: CUDA driver version is insufficient for CUDA runtime version ; GPU=-1 ; hostname=309636a2e4d3 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices);

The OS that Immich Server is running on

Ubuntu 25.04 (ML is on Windows 11)

Version of Immich Server

2.0.1

Version of Immich Mobile App

2.0.1

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

name: immich_remote_ml

services:
    immich-machine-learning:
      container_name: immich_machine_learning
      # Note the `-cuda` at the end
      image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
      # Note the lack of an `extends` section
      deploy:
        resources:
          reservations:
            devices:
              - driver: nvidia
                count: 1
                capabilities:
                  - gpu
      volumes:
        #- "C:\Code\Immich-ML\cache:/cache"
        # ...
        - type: bind
          source: ./cache
          target: /cache
      restart: always
      ports:
        - 3003:3003

Your .env content

n/a

Reproduction steps

1. Upload images to Immich
2. See error in ML console

Relevant log output

2025-10-12 17:48:07.182877091 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 35: CUDA driver version is insufficient for CUDA runtime version ; GPU=-1 ; hostname=309636a2e4d3 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices);

Additional information

No response