DivideByZero error causing plots not to update

[jacobtomlinson]

On systems with shared memory the total device memory returned by pynvml is 0. This results in some DivideByZero errors in places where this value is used.

[E 2025-05-07 14:43:37.537 ServerApp] Exception in callback <bound method GPUResourceWebSocketHandler.send_data of <jupyterlab_nvdashboard.apps.gpu.GPUResourceWebSocketHandler object at 0xf96f511b0170>>
    Traceback (most recent call last):
      File "/home/jtomlinson/miniforge3/envs/rapids-25.04/lib/python3.12/site-packages/tornado/ioloop.py", line 937, in _run
        val = self.callback()
              ^^^^^^^^^^^^^^^
      File "/home/jtomlinson/miniforge3/envs/rapids-25.04/lib/python3.12/site-packages/jupyterlab_nvdashboard/apps/gpu.py", line 113, in send_data
        (stats["gpu_memory_total"] / gpu_mem_sum) * 100, 2
         ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
    ZeroDivisionError: float division by zero

jupyterlab-nvdashboard/jupyterlab_nvdashboard/apps/gpu.py

Line 113 in a71f5f9

(stats["gpu_memory_total"] / gpu_mem_sum) * 100, 2

It would be good to handle this more gracefully. Some graphs just fail to update, while others like the memory usage graph shows 18EB of memory.

In this case we probably need to query the host memory via psutil and display that data instead.

Unfortunately this is only reproducible on machines where GPU memory is being reported by NVML as 0. But if you have such a system you can run the following script.

# memory_mre.py
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "nvidia-ml-py",
# ]
# ///

import pynvml
pynvml.nvmlInit()


print("Detecting GPU memory")
for gpu_idx in range(pynvml.nvmlDeviceGetCount()):
    handle = pynvml.nvmlDeviceGetHandleByIndex(gpu_idx)
    print(f"GPU {gpu_idx}: {pynvml.nvmlDeviceGetMemoryInfo(handle).total} bytes")

[pentschev]

The issue is that nvmlDeviceGetMemoryInfo returns None for any devices that do not have a dedicated memory resource. Similar to Dask-CUDA, I think here the assumption that has been made was that a GPU always has a memory resource, and that assumption now has been made invalid.

The problem with this is that we need to change the paradigm of reporting, the options I see are:

1. Report just host memory;
2. Duplicate memory report, as in both host and device memory will report exactly the same;
3. Create a special case for the user's benefit, which is essentially the same as 1 but phrasing it differently for such systems, like instead of "Host/System Memory" call it something like "Host/System+Devices X,Y,Z Memory" (where X,Y,Z always reduces to 0 at this time);
   i. There's another case I can imagine but don't know if it exists today: multiple GPUs sharing system memory. In this case it's likely that "system memory" has to be partitioned and more specific to "NUMA node memory", as in certain GPU(s) would share memory with only NUMA node 0, and other GPU(s) may share memory with only NUMA node 1, for example. Perhaps in this hypothetical case we would need to break down the notion of system memory for the multiple NUMA nodes, as that becomes relevant from a GPU perspective.

Personally, I'd go for 3 now and ignore 3.i, but I can see an argument of simplifying everything and just go with 1. The option I dislike the most is 2 because that will be too confusing with users thinking they have twice as much memory than they actually have.