[FEATURE REQUEST] Compatibility with LEGACY Nvidia driver 390

[PalasX]

Is your feature request related to a problem? Please describe.
I have a machine with an NVS 4200M Nvidia GPU in it, supported by the Nvidia 390.157 driver.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.157                Driver Version: 390.157                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  NVS 4200M           Off  | 00000000:01:00.0 N/A |                  N/A |
| N/A   53C    P0    N/A /  N/A |      0MiB /   452MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

I can run CUDA 9.1 on it, but not 10 and up. for example, this works:
sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:9.1-base-ubuntu17.04 nvidia-smi

but this does not:
sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:10.0-base-ubuntu14.04 nvidia-smi
with the error:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=10.0, please update your driver to a newer version, or use an earlier cuda container: unknown.

9.2 also fails:
docker run --gpus all nvidia/cuda:9.2-base nvidia-smi

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=9.2, please update your driver to a newer version, or use an earlier cuda container: unknown.
ERRO[0000] error waiting for container: context canceled

Is there any possibility to build this project against CUDA 9.1 for those stuck on the LEGACY driver?

Thanks for any info or response!!! AWESOME project BTW!

[wanjohiryan]

Hi @PalasX

Thank you for reaching out.

By supporting legacy Nvidia GPUs, there is a possibility of breaking Vulkan. |

A while back i tried to use qwantify with Cuda 11.6 and it kept breaking Vulkan

However, give me some time and I will look into this and get back to you.

[ehfd]

Try using cuda-compat perhaps.

[PalasX]

installed cuda-compat, then ran:
docker run --gpus all nvidia/cuda:11.0.3-base nvidia-smi
got:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=11.0, please update your driver to a newer version, or use an earlier cuda container: unknown.
ERRO[0000] error waiting for container:

it appears that the images for anything older than 11.0.3 just arent hosted on dockerhub anymore.

do i need to do something besides installing cuda-compat to use it properly?

[ehfd]

Now, I understand the correct behavior.

https://docs.nvidia.com/deploy/cuda-compatibility/#forward-compatible-upgrade
https://docs.heavy.ai/installation-and-configuration/installation/upgrading-omnisci/cuda-compatibility-drivers

You need to add LD_LIBRARY_PATH=/usr/local/cuda/compat:$LD_LIBRARY_PATH inside the container. If you want to add this environment while starting the container, something like LD_LIBRARY_PATH=/usr/local/cuda/compat:/usr/lib/x86_64-linux-gnu:/usr/lib/i386-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 should work.

I am, however, unsure of whether GeForce GPUs are supported.
Need to test 1070/80, 20xx, 30xx, 40xx.