Adding CUDA 13 support

[jakirkham]

Raising this issue to track adding CUDA 13.

ref: https://developer.nvidia.com/cuda-13-0-0-download-archive

[trivialfis]

#11610 We don't have CI tests or pypi packages yet.

[jakirkham]

Ah thanks for pointing that out Jiaming! 🙏

Looked for CUDA 13, but didn't spot that one

It looks like that is also unreleased. Is that right?

[trivialfis]

Correct. We will need some tests before claiming ctk13 support. Keeping this issue open.

[hcho3]

TODOs

• Rebuild VM images (*) for CI workers to use the latest CUDA driver (preferably the open-source flavor)
• Create Docker images for CUDA 13 with amd64 and arm64
• Add CI pipelines using the CUDA 13 Docker images

(*) Here, VM refers to an Amazon EC2 instance, with access to NVIDIA Tesla T4 card(s).

[hcho3]

Currently, CI workers are using the CUDA driver 565 (legacy flavor):
https://github.com/dmlc/xgboost-devops/blob/ea75e4b680fd8444b40cffbca4c9bcd089368655/vm_images/linux-amd64/install_drivers.sh#L13

CUDA 13 requires driver 580, so I will go ahead and build new VMs with driver 580. I need to ensure that the new VM can host Docker containers with CUDA 12 and 13.

[trivialfis]

Please note that CTK 13 has breaking changes and is not binary compatible with CTK 12.

[hcho3]

Yes, we will need to maintain two separate CI pipelines to build and test XGBoost for CUDA 12 and 13 separately.

[hcho3]

Good news: All existing CI workflows pass when using the 580 driver: #11654. This means that we can use the 580 driver and host the existing CUDA 12 containers. I will go ahead and finalize the new VM image with the 580 driver.

[trivialfis]

That's great news! It would be great if we could add an additional pipeline for single-GPU test that disables HMM with kernel open. I will dig up some notes later for how to.
The HMM allows CUDA kernel to access host memory. This has unwanted effect of hiding invalid memory access.

[trivialfis]

See https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id5 Option 2: Disable HMM for UVM

[hcho3]

@trivialfis Maybe we should disable HMM for all CI jobs?