Nvidia-gpu-operator 25.3.3 / Openshift 4.18.21

[Ali-CTN]

Hi folks,

We are using 4.18.21 in our Openshift environment. When we installed the Nvidia operator and used Nvidia driver version 535.261.03, there was no problem. We installed the new driver to upgrade the CUDA version. The new versions are: Driver Version: 580.82.07 CUDA Version: 13.0. However, there are currently labels indicating issues with mig operations on my nodes:

nvidia.com/gpu.deploy.nvsm: paused-for-mig-change

and the following logs are present in the nvidia-mig-manager pods;

time="2025-09-12T18:54:04Z" level=fatal msg="Error applying MIG configuration with hooks: error initializing NVML: ERROR_LIBRARY_NOT_FOUND"

Is the operator and driver in this version compatible with the relevant OpenShift version?

thanks for the answers.

[rahulait]

Hi @Ali-CTN , we'll need more info on this cluster. How was the driver installed (via package manager or using gpu-operator), how was the upgrade carried out (helm upgrade or just editing cluster-policy and specifying new driver version, or updating via package manager). We will need the must-gather logs to check and see the current state of the cluster. This can be obtained by running the following command on a machine which has kubectl access to the cluster. Please attach them here or submit to us via email: operator_feedback@nvidia.com

curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh
chmod +x must-gather.sh
./must-gather.sh