[HW Accel Support]: Cannot start docker with NVidia device driver

[iamapuzzle]

Describe the problem you are having

When I run sudo docker compose up -d I get error:
prompt:~/Frigate$ sudo docker compose up -d
[+] Running 1/2
✔ Network frigate_default Created 0.1s
⠹ Container frigate Starting 0.2s
Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

I'm trying to enable my NVidia Quadro 3000M GPU for use with Frigate on a Dell M6600 Laptop. In all honesty, I originally ignored my GPU when trialling Frigate as I thought my GPU was too old, but NickM's response to my other question, got me to try it:
#18249 (comment)

I was unable to load the NVidia Drivers for my GPU on Xubuntu 24.04 (black screen after booting), and found out I needed an earlier kernel. Not ideal, but this system will be offline so, I can live with that for now. So I built a new Xubuntu 20.04 system and was able to load the NVidia 390 drivers, which are the latest for this GPU. Yes, I know it's old, but it's all I've got! All info in this post is from this 20.04 install, to ease any confusion.

nvidia-smi outputs the following (sorry, I don't know how to paste this as a fixed width font to preserve the formatting):
`+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.157 Driver Version: 390.157 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro 3000M Off | 00000000:01:00.0 N/A | N/A |
| 0% 54C P0 N/A / N/A | 157MiB / 1985MiB | N/A Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
`

I don't have docker installed as a snap, that was the problem for some other people reporting this error.
I had installed docker from the apt repository following these instructions:
https://docs.docker.com/engine/install/ubuntu/
The hello world docker example runs.

I also tried the instructions on this page (I thought this would be good since it is for Ubuntu 20.04):
https://docs.docker.com/compose/how-tos/gpu-support/
But I get the exact same error:
prompt:~/NVidia$ sudo docker compose up -d
[+] Running 1/2
✔ Network nvidia_default Created 0.1s
⠙ Container nvidia-test-1 Starting 0.1s
Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

I've tried many combinations of NVIDIA_VISIBLE_DEVICES, NVIDIA_DRIVER_CAPABILITIES, device_ids, count and labels, to try and get things working, but I'm at a complete loss. Any suggestions?

sudo docker logs frigate
Doesn't return anything as the container hasn't started. dmesg doesn't report anything during this. syslog reports the error listed above and has networking related things and successful overlay filesystems, but nothing related to graphics that I can see. I don't know how else to get more detailed logs.

I've been though this web page thoroughly:
https://docs.frigate.video/configuration/hardware_acceleration
One potential source of confusion is that there appears to be both a Quadro 3000M and a Quadro M3000M. Mine is the GF104GLM Quadro 3000M. I think the M3000M is newer. As mentioned, initially I thought mine was just too old and unsupported, but when challenged to try it, I'm giving it my best shot!

Version

Can't get to WebUI, but it's a recent download

Frigate config file

Frigate hasn't loaded yet, so I don't think this is relevant to this issue.  But my Frigate config is unchanged from my last post:
https://github.com/blakeblackshear/frigate/discussions/18249

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    stop_grace_period: 30s
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "128mb" # updated for my cameras
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              #device_ids: ['0']
              #count: 1
              capabilities: [gpu]
    volumes:
      - ./config:/config
      - ./certificates:/etc/letsencrypt/live/frigate:ro
      - ./storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "443:8971"
      - "8554:8554" # RTSP feeds

Relevant Frigate log output

prompt:~/Frigate$ sudo docker images
REPOSITORY                        TAG                       IMAGE ID       CREATED         SIZE
ghcr.io/blakeblackshear/frigate   stable                    79c58d895643   5 weeks ago     2.71GB
nvidia/cuda                       12.3.1-base-ubuntu20.04   d13839a3c4fb   17 months ago   246MB
prompt:~/Frigate$ sudo docker compose up -d
[+] Running 1/2
 ✔ Network frigate_default  Created                                                            0.1s 
 ⠹ Container frigate        Starting                                                           0.2s 
Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

I did have the hello world docker image there too earlier, but during testing, I purged everything and haven't put it back.

Relevant go2rtc log output

N/A

FFprobe output from your camera

N/A

Install method

Docker Compose

Object Detector

CPU (no coral)

Network connection

Wired

Camera make and model

N/A

Screenshots of the Frigate UI's System metrics pages

Can't get to WebUI

Any other information that may be helpful

docker-compose.yml file for NVidia Test:

services:
test:
image: nvidia/cuda:12.3.1-base-ubuntu20.04
command: nvidia-smi
environment:
NVIDIA_VISIBLE_DEVICES: all

NVIDIA_DRIVER_CAPABILITIES: compute,utility

NVIDIA_DRIVER_CAPABILITIES: compute,video,utility

  NVIDIA_DRIVER_CAPABILITIES: all
labels:
  - com.nvidia.volumes.needed="nvidia_driver"
deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          #count: 1
          capabilities: [gpu]

[NickM-27]

One potential source of confusion is that there appears to be both a Quadro 3000M and a Quadro M3000M. Mine is the GF104GLM Quadro 3000M. I think the M3000M is newer. As mentioned, initially I thought mine was just too old and unsupported, but when challenged to try it, I'm giving it my best shot!

yes, it does appear that the GF104GLM does not support the necessary compute capability, that is very confusing and sorry for the goose chase there

[iamapuzzle]

Thank you Nick. At least I tried getting it working. I think I best bet is to add a Coral for detections now.