[Feature]: Make Taichi support Cuda 12.X

[Paulo-21]

What feature or enhancement are you proposing?

Hello,
Since Genesis is heavily based on taichi, I noticed that it's based on cuda 10.0 (Sept 2018) with this code :

import taichi as ti
ti.init(arch=ti.cuda)
print("CUDA version :", ti._lib.core.cuda_version())

I think we could have a huge benefit by patching Taichi with a support for newer version of Cuda, like 12.x.

Motivation

I have a RTX 6000 (Ada) whitch is at least 10 times faster than my personnal 1070 GTX.
But when i run Genesis, the simulation is only 1.9x faster on the RTX 6000.

Potential Benefit

i saw that taichi use llvm 15 and cuda to generate GPU code,
Having a version of taichi with llvm 20 and cuda 12.x could have a huge benefit for user with recent graphics Card.
In the release note of cuda we can see than there is some update about routine in cuSolver and cuBLAS and in the nvidia compiler.
I think llvm 15 to 20 have receive some improvement too.
I cannot quantify the benefit but i think that by improving the way we compile and generate kernel code we can have a performance improvement.

What is the expected outcome of the implementation work?

Having better gpu code generation with a better use of the capabilities of ours novel GPU and newest research about routines from nvidia.
Genesis can be faster IMO.

Additional information

I have started to investigate and there a PR where someone started to patch Taichi to use LLVM 20.
When i tried to compile taichi with cuda-toolkit 12.9, with the docs informations I have some error about cuda function name that aren't reconized.
I have tried every version of cuda toolkit to build taichi with (12.9 -> 12.4 -> 11.8 -> 11.4 -> 11.1 -> 10.0 (normaly shipped with taichi))
but none of it worked and it seems that it's because someone in the last commit of this file used some preview cuda API who has never been released like this.

Best Regards,
Paul

[Kashu7100]

I am using cuda 12.9 on 5090 - no problem.
Note that there is a know bug in nvidia driver (Genesis-Embodied-AI/Genesis#1157)

[Paulo-21]

Thank for the attention and the respect you give to my issue.

"I am using cuda 12.9 on 5090 - no problem."
Ok cool and so what ? You have installed cuda toolkit 12.9 on you computer with the last driver ?
Nice, me too.

Or you have successfully compile taichi from source with the flag CUDA_TOOLKIT enable and using cuda 12.9 for the compilation ?

I think you miss undestund what i was talking about.
I wasn't talking about crash or issue about run genesis with taichi on recent graphics card.
I was talking about the code generated by taichi for the GPU, if it can be more efficient.

I have the last driver with cuda 12.9 like you but taichi did not use the cuda version installed on your machine.
It use it s own.

if taichi were used the cuda version installed on the machine i will not assume that the creator of taichi will say it will add support for cuda 12.4 some day. In that case you just have to install the cuda 12.4 driver and it's works.
taichi-dev/taichi#8506 (comment)

So i think taichi use it's own cuda version, something close to the 10.0 version.
I read the documentation correclty from cuda, you need a traditionnal c++ compiler to compile host code (the c++ part) and the cuda compiler then it compile some binary with PTX assemlby which is an intermediare assembly code for the GPU. It put every thing into the final binarie.

When it execute the binarie the Jit compiler from the driver compile the PTX assembly into the GPU assembly code so.

It's like pytorch that ship torch with cuda. There a lot of api that are deprecate with the time so that whyb the cuda runtime is shiped with the binairie and the jit compiler from the driver just compile it for the host architeture.
If it not do that you can write code that use an api that are stable in the 11.8 version of cuda and deprecated and removed in earlier version for exemple.
Imagine that the user use your librairi and update his drivers. and it's cuda toolkit.
the code will be broken ? or he have to manage PATH and LD Variable all the time.

And if the cuda librairie was dynamicly linked i will be able to see libcudart with this command :
lsof -p | grep cuda
But no.

Well if you want more information maybe you can try to read the cuda documentation.
https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compilation-workflow

And they use llvm 15 which it kind of old, you can see it by just printing the debug information of taichi and it's in the source code too.

You can test by your self the cuda version of taichi by testing this little code :

import taichi as ti
ti.init(arch=ti.cuda)
print("CUDA version :", ti._lib.core.cuda_version()

I'm just posting that here rather than on the taichi repo because it's seems to be dead, and Genesis is heavily based on it and very popular at this moment, i was thinking that it can found more people motivated to port the code to cuda 12.X here rather than in the taichi repo.

"Note that there is a know bug in nvidia driver (#1157)"
Ok, but nothing related to what i'm talking about.

[Kashu7100]

Sorry, I think I wasn't getting your point. I think @hughperkins can add more on this.

[hughperkins]

Hi @Paulo-21 , thanks! Yes, we probably want to upgrade llvm soon-ish.

Interesting observation about ti._lib.core.cuda_version(). Note that looking through the code, it looks like this is simply an arbitrary string, hard-coded to 10 here? https://github.com/taichi-dev/taichi/blob/a1d99c9c4cef9de5f9b633a515d227677ffab9a0/cmake/TaichiCore.cmake#L99

• this is then returned from https://github.com/taichi-dev/taichi/blob/a1d99c9c4cef9de5f9b633a515d227677ffab9a0/taichi/common/core.cpp#L88 I believe
• which is exported to the python here https://github.com/taichi-dev/taichi/blob/a1d99c9c4cef9de5f9b633a515d227677ffab9a0/taichi/python/export_misc.cpp#L133 I think?

When it execute the binarie the Jit compiler from the driver compile the PTX assembly into the GPU assembly code so.

This is exactly right. Yes!

I read the documentation correclty from cuda, you need a traditionnal c++ compiler to compile host code (the c++ part) and the cuda compiler then it compile some binary with PTX assemlby which is an intermediare assembly code for the GPU. It put every thing into the final binarie.

So, as you mentioned above, Taichi is using JIT. And the input to the JIT is: LLVM IR code.

• the LLVM IR code is converted to ptx, using the LLVM library here https://github.com/taichi-dev/taichi/blob/a1d99c9c4cef9de5f9b633a515d227677ffab9a0/taichi/runtime/cuda/jit_cuda.cpp#L76-L240

You are right that version LLVM 15 generates relatively old ptx, and it could be interesting to evaluate performance with a newer version of LLVM, when using newer cards.

re: taichi-dev/taichi#8735

AFAIK @johnnynunez 's branch should allow you to run using LLVM 20, which will generate newer PTX, as you allude to.

Note that the cuda toolkit is AFAIK NOT used for building the taichi kernels, or for building the taichi source code, in order to buil the taichi kernels. Per my undrestanding, the only thing you need is: LLVM. And you can turn off the cuda toolkit option.

These are the options I think you should likely be able to build with:

Per my understanding, CUDA toolkit is only used to give access to something called 'cupti', which looks like it migth be used for profiling? eg see. https://github.com/taichi-dev/taichi/blob/a1d99c9c4cef9de5f9b633a515d227677ffab9a0/taichi/rhi/cuda/cupti_toolkit.cpp#L6

If you manage to get this working, would be super interetsed in any performance benchmarking results you can provide. Please consider posting them into this thread. That would be awesome 🙌

[johnnynunez]

What feature or enhancement are you proposing?

Hello, Since Genesis is heavily based on taichi, I noticed that it's based on cuda 10.0 (Sept 2018) with this code :

import taichi as ti
ti.init(arch=ti.cuda)
print("CUDA version :", ti._lib.core.cuda_version())

I think we could have a huge benefit by patching Taichi with a support for newer version of Cuda, like 12.x.

Motivation

I have a RTX 6000 (Ada) whitch is at least 10 times faster than my personnal 1070 GTX. But when i run Genesis, the simulation is only 1.9x faster on the RTX 6000.

Potential Benefit

i saw that taichi use llvm 15 and cuda to generate GPU code, Having a version of taichi with llvm 20 and cuda 12.x could have a huge benefit for user with recent graphics Card. In the release note of cuda we can see than there is some update about routine in cuSolver and cuBLAS and in the nvidia compiler. I think llvm 15 to 20 have receive some improvement too. I cannot quantify the benefit but i think that by improving the way we compile and generate kernel code we can have a performance improvement.

What is the expected outcome of the implementation work?

Having better gpu code generation with a better use of the capabilities of ours novel GPU and newest research about routines from nvidia. Genesis can be faster IMO.

Additional information

I have started to investigate and there a PR where someone started to patch Taichi to use LLVM 20. When i tried to compile taichi with cuda-toolkit 12.9, with the docs informations I have some error about cuda function name that aren't reconized. I have tried every version of cuda toolkit to build taichi with (12.9 -> 12.4 -> 11.8 -> 11.4 -> 11.1 -> 10.0 (normaly shipped with taichi)) but none of it worked and it seems that it's because someone in the last commit of this file used some preview cuda API who has never been released like this.

Best Regards, Paul

im working on it, please be patiend. I just joined nvidia, a lot of training first two weeks

[johnnynunez]

Maybe I upgrade it to LLVM21 also, to support newer devices, LLVM20 is not compatible with Thor, Spark and GB300 yet

[johnnynunez]

im going to start today making PRs @hughperkins @Paulo-21 @duburcqa

I count around 10-12

[hughperkins]

As far as "cuda toolkit 12", digging a little:

• CUDA Toolkit 12.8 provides libdevice.10.bc bytecode file, which contains ~356 exported functions:

$ find /usr/local/cuda-12.8/ -name '*.bc'
/usr/local/cuda-12.8/nvvm/libdevice/libdevice.10.bc
$ ~/.cache/ti-build-cache/llvm15/bin/llvm-nm /usr/local/cuda/nvvm/libdevice/libdevice.10.bc | awk '{print $3}' | wc -l
356

• this has been shrunk slightly, into slim_libdevice bytecode file, in gstaichi repo:

$ find . -name '*device*.bc'
./external/cuda_libdevice/slim_libdevice.10.bc
./_skbuild/linux-x86_64-3.10/cmake-install/python/gstaichi/_lib/runtime/slim_libdevice.10.bc
./_skbuild/linux-x86_64-3.10/setuptools/lib.linux-x86_64-cpython-310/gstaichi/_lib/runtime/slim_libdevice.10.bc

• which has ~320 exported functions:

$ ~/.cache/ti-build-cache/llvm15/bin/llvm-nm ./external/cuda_libdevice/slim_libdevice.10.bc | awk '{print $3}' | wc -l
320

So, then a question is: why does cuda toolkit 12.8 contains libdevice.10.bc? I asked DeepSeek. DeepSeek said:

---

"The version number in the filename (e.g., libdevice.10.bc) does NOT refer to the CUDA Toolkit version. Instead, it refers to the compute capability (architecture version) that the library is designed to support as a minimum and for which it provides the most fundamental implementations.

"Here’s a breakdown of the reasons and the logic behind the naming:

1. It's a "SM Version" Identifier, Not a "CUDA Version" Identifier

The primary role of libdevice is to provide mathematical functions (like sin, exp, sqrt) for the GPU (the "device"). The implementation of these functions can be optimized for different NVIDIA GPU architectures (known as compute capabilities or SM versions, e.g., SM 5.0, SM 7.0, SM 8.0, SM 9.0).

The number in the filename signifies the oldest compute capability that this specific library version primarily targets. libdevice.10.bc is the version that supports compute capability 3.0 and above (the number is a coded reference, explained next).

2. The Encoding Scheme

The number is encoded. You can decode it to find the minimum compute capability:

• libdevice.10.bc: The number 10 is a decimal representation of the compute capability multiplied by 10.
  • 10 / 10 = 1.0
  • This library is the base version, designed to work for the foundational SM 3.0 (Kepler) architecture and all later ones. The "1.0" is a historical baseline.

NVIDIA also provides other versions for compatibility or testing:

• libdevice.compute_XX.YY.bc: These are older, more specific versions (e.g., compute_50, compute_61). The libdevice.10.bc is the modern, unified replacement for these.

3. Why libdevice.10.bc is Used in CUDA 12.x

The toolchain in CUDA 12.8 is sophisticated enough to take the single, unified libdevice.10.bc file and internally link and optimize it for your specific target architecture.

Here's how it works:

1. You compile your code with -arch=sm_89 (for an RTX 40-series card).
2. The NVVM compiler (which uses LLVM under the hood) links against libdevice.10.bc.
3. During the linking and optimization process, the compiler knows the target is sm_89. It takes the generic implementations in libdevice.10.bc and aggressively optimizes them specifically for the SM 8.9 architecture, often generating much more efficient code than a generic pre-compiled library could.

In essence, libdevice.10.bc contains the "source code" for the math functions in LLVM IR form, and the compiler finishes the job of tailoring it to your GPU.

Summary

file	description
libdevice.10.bc	This is the unified, modern math library in LLVM bitcode format. The 10 indicates it serves as the foundation for compute capability 3.0 and higher. It is optimized at compile-time for your specific -arch target.
CUDA Toolkit 12.8	This is the version of the host-side compiler, tools, and drivers you are using. It includes the latest libdevice library and the smart compiler that knows how to use it.
"

(End of DeepSeek output)

---

I will probably

• get rid of the cuda_version() function, which is misleading
• internally, rename CUDA_VERSION to LIBDEVICE_VERSION

[hughperkins]

"Make Taichi support CUDA 12.x" addressed in #202

[hughperkins]

As far as upgrading LLVM to 20 or so, which I think is the key problem you are trying to solve?, there is an existing issue for this at #86, so I intend to close this current issue (166), if/when #202 is merged. Unless you have any major objections/questions/comments/concerns, @Paulo-21 ?