AMD GPU Partition Pre-requisites

[viswar4]

With AMD GPU operator v1.3.0 there is no mention of a specific RoCM Version which is needed. Is the GPU partitioning feature supported with ROCM 6.2 for example or needs 6.4 minimum?

[yansun1996]

Hi @viswar4 , please use ROCm 6.4 or later driver (amdgpu-dkms driver 6.12.12 or later) to support GPU partitioning, the Device Config Manager (DCM) image in release v1.3.0 is already packaged with ROCm 6.4 runtime libraries to let you configure the GPU partitioning.

This is a good point, since we're providing more support on various feature and operands (device plugin, node labeller, metrics exporter, device test runner, device config manager) we need to give a compatibility of our features with corresponding amdgpu-dkms driver versions and ROCm runtime library versions.

[viswar4]

Hi Yansun,

Thanks for the confirmation. I have one more concern. When we are running multiple pods which request partitioned gpus we face issues. If we only run 1 pod on that worker node every thing is ok. Do we need to manually track gpu ids and ensure no 2 pods request same gpu.
Something like HIP_VISIBLE_DEVICES?
or is it because of ROCm version only?

[yansun1996]

Hi @viswar4 , as for scheduling the workload, we're utilizing the Kubernetes Device Plugin, our implementation is in this repo. When you install and configure the AMD GPU Operator you could see the device plugin pod is running and reporting amd.com/gpu as resource. (More advanced device plugin config is in this doc https://instinct.docs.amd.com/projects/gpu-operator/en/latest/device_plugin/device-plugin.html)

After that when you try to run the workload pod, your pod could request amd.com/gpu resource and the scheduler would automatically schedule available GPUs to the new workload pod, you don't have to track the ID and worry that one GPU got assigned to multiple workload pods.