is it possible to reload nvidia kernel modules if they were unloaded?

[tan-wei-xin-alez]

so a follow up to how to run nvidia-bug-report or equivalent, the approach there worked because we had already rebooted the node but we have another node with the same error

10.5.0.81: kern: warning: [2025-10-30T18:49:51.987704164Z]: NVRM: Attempting to remove device 0000:84:00.0 with non-zero usage count!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817484164Z]: NVRM: GPU at PCI:0000:84:00: GPU-b5388638-ffe9-8a2b-1084-b314d92138de
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817502164Z]: NVRM: GPU Board Serial Number: 1792525056648
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817504164Z]: NVRM: Xid (PCI:0000:84:00): 79, GPU has fallen off the bus.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817515164Z]: NVRM: GPU 0000:84:00.0: GPU has fallen off the bus.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817517164Z]: NVRM: GPU 0000:84:00.0: GPU serial number is 1792525056648.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817525164Z]: NVRM: kgspRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817547164Z]: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817570164Z]: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817580164Z]: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817589164Z]: NVRM: prbEncStartAlloc: Can't allocate memory for protocol buffers.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817593164Z]: NVRM: A GPU crash dump has been created. If possible, please run
NVRM: nvidia-bug-report.sh as root to collect this data before
NVRM: the NVIDIA kernel module is unloaded.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817668164Z]: NVRM: Xid (PCI:0000:84:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817967164Z]: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817994164Z]: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:274
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818038164Z]: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818043164Z]: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:274
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818051164Z]: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818055164Z]: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:274
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818103164Z]: NVRM: nvAssertFailedNoLog: Assertion failed: status == NV_OK @ journal.c:2266
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649505164Z]: {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 516
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649524164Z]: {1}[Hardware Error]: It has been corrected by h/w and requires no further action
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649529164Z]: {1}[Hardware Error]: event severity: corrected
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649534164Z]: {1}[Hardware Error]:  Error 0, type: corrected
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649538164Z]: {1}[Hardware Error]:  fru_text: PcieError
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649543164Z]: {1}[Hardware Error]:   section_type: PCIe error
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649546164Z]: {1}[Hardware Error]:   port_type: 6, downstream switch port
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649550164Z]: {1}[Hardware Error]:   version: 0.2
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649553164Z]: {1}[Hardware Error]:   command: 0x0407, status: 0x0010
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649558164Z]: {1}[Hardware Error]:   device_id: 0000:82:01.0
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649562164Z]: {1}[Hardware Error]:   slot: 3
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649564164Z]: {1}[Hardware Error]:   secondary_bus: 0x84
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649568164Z]: {1}[Hardware Error]:   vendor_id: 0x1000, device_id: 0xc030
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649571164Z]: {1}[Hardware Error]:   class_code: 060400
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649574164Z]: {1}[Hardware Error]:   bridge: secondary_status: 0x0000, control: 0x0002
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649579164Z]: {1}[Hardware Error]:   aer_cor_status: 0x00000001, aer_cor_mask: 0x00002000
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649583164Z]: {1}[Hardware Error]:   aer_uncor_status: 0x00000020, aer_uncor_mask: 0x00000000
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649586164Z]: {1}[Hardware Error]:   aer_uncor_severity: 0x07ef6030
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649590164Z]: {1}[Hardware Error]:   TLP Header: 00000000 00000000 00000000 00000000

this time though, the same approach does not work as we get the following error

➜  ~ kubectl run "node-debugger-${USER}" --restart=Never --namespace kube-system --image nvcr.io/nvidia/cuda:12.5.0-base-ubuntu22.04 --overrides '{"spec": {"runtimeClassName": "nvidia", "nodeSelector": {"kubernetes.io/hostname": "<node_name>"}}}' -- bash -c 'sleep 99999' && \
  kubectl debug "node-debugger-${USER}" -it --namespace kube-system --profile sysadmin --image nvcr.io/nvidia/cuda:12.8.0-base-ubuntu22.04 --target "node-debugger-${USER}"; \
  kubectl delete pod "node-debugger-${USER}" --namespace kube-system --force
pod "perception-train-test-<user>" deleted
pod perception-jobs/perception-train-test-<user> terminated (StartError)
failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: detection error: nvml error: unknown error

we can get around it by removing "runtimeClassName": "nvidia" but obviously, the error is popping up because the kernel modules for nvidia have been unloaded

➜  ~ kubectl run "node-debugger-${USER}" --restart=Never --namespace kube-system --image nvcr.io/nvidia/cuda:12.8.0-base-ubuntu22.04 --overrides '{"spec": {"nodeSelector": {"kubernetes.io/hostname": "<node_name>"}}}' -- bash -c 'sleep 99999' && \
  kubectl debug "node-debugger-${USER}" -it --namespace kube-system --profile sysadmin --image nvcr.io/nvidia/cuda:12.8.0-base-ubuntu22.04 --target "node-debugger-${USER}"; \
  kubectl delete pod "node-debugger-${USER}" --namespace kube-system --force
pod/node-debugger-<user> created
Targeting container "node-debugger-<user>". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-9srlb.
If you don't see a command prompt, try pressing enter.
root@node-debugger-<user>:/# apt-get update && apt-get install -y kmod
...
root@node-debugger-<user>:/# lsmod | grep nvidia
root@node-debugger-alez:/# modprobe nvidia
modprobe: FATAL: Module nvidia not found in directory /lib/modules/6.12.45-talos

Since Talos is an immutable OS, I guess there is no way to reload the nvidia kernel modules if the above happens without rebooting the machine, am I correct? Or is there some really hacky way to do it? Because we have other pods still running on the node that can reach the other GPUs on the node (although they will experience an error with nvml if they try to use them), it's just that this one GPU experienced this error and unloaded the nvidia kernel modules in the process

[rothgar]

The Talos API is the only process allowed to load and unload kernel modules so you'll have to submit a patch to the API to load the kernel modules.

machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset
  sysctls:
    net.core.bpf_jit_harden: 1

I'm not sure if this requires a reboot

[tan-wei-xin-alez]

oh so theoretically I can remove the already existing nvidia kernel modules via talos_machine_configuration_apply in terraform then add them back in again and it should work? Since it uses the same API as talosctl apply-config, it should be same right? Or do I have to specifically use talosctl machineconfig patch or talosctl patch (I assumed they were all using the same API even though the commands were different but I may be mistaken)

I'm not sure if this requires a reboot

well if this is required regardless then the above is moot, but nice to know the Talos API is supposed to be able to load and unload kernel modules even without a reboot

[frezbo]

yes, you'd apply a patch to remove then reboot, currently it doesn't unload

[tan-wei-xin-alez]

yes, you'd apply a patch to remove then reboot, currently it doesn't unload

ah so if the kernel modules were already unloaded, then technically applying a patch should load them even without rebooting? Ok, I'll give that a shot