ti.init(arch=ti.cuda) error with [cuda_driver.h:operator()@92] CUDA Error : while calling init (cuInit)

[xukez2]

Describe the bug
After installing taichi can calling ti.init(arch=ti.cuda), i got the following error msg

[E 07/25/23 21:32:02.308 106374] [cuda_driver.h:operator()@92] CUDA Error �_:  while calling init (cuInit)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kexu/miniconda3/lib/python3.10/site-packages/taichi/lang/misc.py", line 460, in init
    impl.get_runtime().create_program()
  File "/home/kexu/miniconda3/lib/python3.10/site-packages/taichi/lang/impl.py", line 376, in create_program
    self.prog = _ti_core.Program()
RuntimeError
Taichi CUDA test failed: Command '['/home/kexu/miniconda3/bin/python', '-c', 'import taichi as ti; ti.init(arch=ti.cuda)']' returned non-zero exit status 1.

Log/Screenshots
Running ti diagnose i got

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy



import: <module 'taichi' from '/home/kexu/miniconda3/lib/python3.10/site-packages/taichi/__init__.py'>

cc: False
cpu: True
metal: False
opengl: True
cuda: True
vulkan: True

OpenGL version 4.6.0 NVIDIA 535.54.03 is supported
GL_ARB_compute_shader:                                         OK 
GL_ARB_gpu_shader_int64:                                       OK 
GL_NV_shader_atomic_float:                                     OK 
GL_NV_shader_atomic_float64:                                   OK 
GL_NV_shader_atomic_int64:                                     OK 

Tue Jul 25 21:32:01 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 ...    On  | 00000000:0B:00.0  On |                  N/A |
| 18%   45C    P0              49W / 250W |   2020MiB /  8192MiB |     13%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1747      G   /usr/lib/xorg/Xorg                         1151MiB |
|    0   N/A  N/A      1937      G   /usr/bin/gnome-shell                        164MiB |
|    0   N/A  N/A      3413      G   ...8687046,15544737911859752941,262144      313MiB |
|    0   N/A  N/A     68276      G   ...sion,SpareRendererForSitePerProcess      218MiB |
|    0   N/A  N/A     80236      G   ...ures=SpareRendererForSitePerProcess      104MiB |
+---------------------------------------------------------------------------------------+

[Taichi] version 1.6.0, llvm 15.0.4, commit f1c6fbbd, linux, python 3.10.12

[Taichi] version 1.6.0, llvm 15.0.4, commit f1c6fbbd, linux, python 3.10.12
[Taichi] Starting on arch=x64

[Taichi] version 1.6.0, llvm 15.0.4, commit f1c6fbbd, linux, python 3.10.12
[Taichi] Starting on arch=opengl

[E 07/25/23 21:32:02.308 106374] [cuda_driver.h:operator()@92] CUDA Error �_:  while calling init (cuInit)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kexu/miniconda3/lib/python3.10/site-packages/taichi/lang/misc.py", line 460, in init
    impl.get_runtime().create_program()
  File "/home/kexu/miniconda3/lib/python3.10/site-packages/taichi/lang/impl.py", line 376, in create_program
    self.prog = _ti_core.Program()
RuntimeError
Taichi CUDA test failed: Command '['/home/kexu/miniconda3/bin/python', '-c', 'import taichi as ti; ti.init(arch=ti.cuda)']' returned non-zero exit status 1.
[Taichi] version 1.6.0, llvm 15.0.4, commit f1c6fbbd, linux, python 3.10.12

Additional comments
pytorch/pycuda is working fine

[houkensjtu]

Hi! Looks like it's an error caused by CUDA driver. Just to make sure, are you working on a native Linux machine, or WSL / VirtualBox?

[xukez2]

hi @houkensjtu, thanks for the reply. I'm working on a native ubuntu. If that's related to the CUDA driver, do you have a recommendation for the driver version?

[houkensjtu]

@xukez2 Both your GPU and your CUDA version looks fine, I don't have a specific recommendation. Can you run ti example? Maybe you should also try re-installing the CUDA driver.

[jim19930609]

Looks like an issue with CUDA Driver, can you verify if your GPU works with other frameworks like Pytorch-GPU?

[xukez2]

@jim19930609 i can use pytorch gpu. I'm trying to have different cuda version to see if that can work

[xukez2]

@jim19930609 and @houkensjtu i downgraded my nvidia driver and cuda to nvidia-driver-525 and cuda 12.0 and it works. Thanks for the help.