how to run nvidia-bug-report or equivalent

[tan-wei-xin-alez]

Hello, one of our nodes lost connection to its nvidia gpu and we got the following error at the kernel level

10.5.0.81: kern: warning: [2025-10-30T18:49:51.987704164Z]: NVRM: Attempting to remove device 0000:84:00.0 with non-zero usage count!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817484164Z]: NVRM: GPU at PCI:0000:84:00: GPU-b5388638-ffe9-8a2b-1084-b314d92138de
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817502164Z]: NVRM: GPU Board Serial Number: 1792525056648
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817504164Z]: NVRM: Xid (PCI:0000:84:00): 79, GPU has fallen off the bus.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817515164Z]: NVRM: GPU 0000:84:00.0: GPU has fallen off the bus.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817517164Z]: NVRM: GPU 0000:84:00.0: GPU serial number is 1792525056648.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817525164Z]: NVRM: kgspRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817547164Z]: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817570164Z]: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817580164Z]: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817589164Z]: NVRM: prbEncStartAlloc: Can't allocate memory for protocol buffers.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817593164Z]: NVRM: A GPU crash dump has been created. If possible, please run
NVRM: nvidia-bug-report.sh as root to collect this data before
NVRM: the NVIDIA kernel module is unloaded.
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817668164Z]: NVRM: Xid (PCI:0000:84:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817967164Z]: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.817994164Z]: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:274
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818038164Z]: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818043164Z]: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:274
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818051164Z]: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78!
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818055164Z]: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:274
10.5.0.81: kern: warning: [2025-10-30T18:49:52.818103164Z]: NVRM: nvAssertFailedNoLog: Assertion failed: status == NV_OK @ journal.c:2266
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649505164Z]: {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 516
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649524164Z]: {1}[Hardware Error]: It has been corrected by h/w and requires no further action
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649529164Z]: {1}[Hardware Error]: event severity: corrected
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649534164Z]: {1}[Hardware Error]:  Error 0, type: corrected
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649538164Z]: {1}[Hardware Error]:  fru_text: PcieError
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649543164Z]: {1}[Hardware Error]:   section_type: PCIe error
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649546164Z]: {1}[Hardware Error]:   port_type: 6, downstream switch port
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649550164Z]: {1}[Hardware Error]:   version: 0.2
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649553164Z]: {1}[Hardware Error]:   command: 0x0407, status: 0x0010
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649558164Z]: {1}[Hardware Error]:   device_id: 0000:82:01.0
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649562164Z]: {1}[Hardware Error]:   slot: 3
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649564164Z]: {1}[Hardware Error]:   secondary_bus: 0x84
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649568164Z]: {1}[Hardware Error]:   vendor_id: 0x1000, device_id: 0xc030
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649571164Z]: {1}[Hardware Error]:   class_code: 060400
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649574164Z]: {1}[Hardware Error]:   bridge: secondary_status: 0x0000, control: 0x0002
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649579164Z]: {1}[Hardware Error]:   aer_cor_status: 0x00000001, aer_cor_mask: 0x00002000
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649583164Z]: {1}[Hardware Error]:   aer_uncor_status: 0x00000020, aer_uncor_mask: 0x00000000
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649586164Z]: {1}[Hardware Error]:   aer_uncor_severity: 0x07ef6030
10.5.0.81: kern: warning: [2025-10-30T18:49:53.649590164Z]: {1}[Hardware Error]:   TLP Header: 00000000 00000000 00000000 00000000

I was wondering how would one go about running nvidia-bug-report.sh with talosctl since there is obviously no way to directly run the command on the node

Would talosctl support contain the information nvidia-bug-report.sh would generate?

[rothgar]

The proper way is probably to run a kubernetes debug pod with the script and run it. You can find details on how to do it here https://www.siderolabs.com/blog/how-to-ssh-into-talos-linux/

[tan-wei-xin-alez]

ok so this wasn't the whole picture but it pointed me in the right direction, here's how one could do it using kubectl debug

➜  ~ kubectl run "node-debugger-${USER}" --restart=Never --namespace kube-system --image nvcr.io/nvidia/cuda:12.8.0-base-ubuntu22.04 --overrides '{"spec": {"runtimeClassName": "nvidia", "nodeSelector": {"kubernetes.io/hostname": "<node_name>"}}}' -- bash -c 'sleep 99999' && \
  kubectl debug "node-debugger-${USER}" -it --namespace kube-system --profile sysadmin --image nvcr.io/nvidia/cuda:12.8.0-base-ubuntu22.04 --target "node-debugger-${USER}"; \
  kubectl delete pod "node-debugger-${USER}" --namespace kube-system --force
pod/node-debugger-<user> created
Targeting container "node-debugger-<user>". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-9vlpg.
If you don't see a command prompt, try pressing enter.
root@node-debugger-<user>:/# nvidia-smi --query-gpu=driver_version --format=csv,noheader
570.172.08
570.172.08
570.172.08
570.172.08
570.172.08
570.172.08
570.172.08
570.172.08
root@node-debugger-<user>:/# apt-get update && apt-get install -y curl
...
root@node-debugger-<user>:/# curl -O https://us.download.nvidia.com/XFree86/Linux-x86_64/570.172.08/NVIDIA-Linux-x86_64-570.172.08.run
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  358M  100  358M    0     0  47.6M      0  0:00:07  0:00:07 --:--:-- 54.1M
root@node-debugger-<user>:/# sh NVIDIA-Linux-x86_64-570.172.08.run --extract-only
Creating directory NVIDIA-Linux-x86_64-570.172.08
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 570.172.08.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
root@node-debugger-<user>:/# cp NVIDIA-Linux-x86_64-570.172.08/nvidia-bug-report.sh /usr/bin/
root@node-debugger-<user>:/# nvidia-bug-report.sh

nvidia-bug-report.sh will now collect information about your
system and create the file 'nvidia-bug-report.log.gz' in the current
directory.  It may take several seconds to run.  In some
cases, it may hang trying to capture data generated dynamically
by the Linux kernel and/or the NVIDIA kernel module.  While
the bug report log file will be incomplete if this happens, it
may still contain enough data to diagnose your problem.

If nvidia-bug-report.sh hangs, consider running with the --safe-mode
and --extra-system-data command line arguments.

Please include the 'nvidia-bug-report.log.gz' log file when reporting
your bug via the NVIDIA Linux forum (see forums.developer.nvidia.com)
or by sending email to '[email protected]'.

By delivering 'nvidia-bug-report.log.gz' to NVIDIA, you acknowledge
and agree that personal information may inadvertently be included in
the output.  Notwithstanding the foregoing, NVIDIA will use the
output only for the purpose of investigating your reported issue.

Running nvidia-bug-report.sh... complete.


Summary of Skipped Sections:

Skipped Component                   | Details
================================================================================
ldd output                          | glxinfo not found
--------------------------------------------------------------------------------
lspci output                        | lspci or hexdump not found
--------------------------------------------------------------------------------
lsusb output                        | lsusb not found
--------------------------------------------------------------------------------
dmidecode output                    | dmidecode not found
--------------------------------------------------------------------------------
modinfo output                      | modinfo not found
--------------------------------------------------------------------------------
xrandr output                       | xrandr not found
--------------------------------------------------------------------------------
vulkaninfo output                   | vulkaninfo not found
--------------------------------------------------------------------------------
ibstat output                       | ibstat not found
--------------------------------------------------------------------------------
acpidump output                     | acpidump not found
--------------------------------------------------------------------------------
mst output                          | mst not found
--------------------------------------------------------------------------------
nvlsm-bug-report.sh output          | nvlsm-bug-report.sh not found
--------------------------------------------------------------------------------

Summary of Errors:

Error Component                     | Details                                                      | Resolution
=========================================================================================================================

then copying out the generated nvidia-bug-report.log.gz can be done with

➜  ~ kubectl get pod "node-debugger-${USER}" --namespace kube-system --output jsonpath='{.spec.containers[*].name}{" "}{.spec.ephemeralContainers[*].name}{" "}'
node-debugger-<user> debugger-9vlpg
➜  ~ kubectl cp "kube-system/node-debugger-${USER}":/nvidia-bug-report.log.gz ./nvidia-bug-report.log.gz --container debugger-9vlpg
tar: Removing leading `/' from member names
➜  ~ ls nvidia-bug-report.log.gz
nvidia-bug-report.log.gz