AWS EFA GPUDirect RDMA validation

[erezzarum]

UCX on AWS EFA instances validate the presence of efa_nv_peermem module to detect if GPUDirect RDMA is supported.
The efa_nv_peermem module might not always be available (depends on OS/AMI), yet the instance does support GPUDirect RDMA.

1. It looks like UCX doesn't make use of this module but uses dmabuf
2. If 1 is true, can we please remove this dependency and find other ways to detect if GPUDirect RDMA is supported? we can make this configurable via environment variable if needed for now.

[brminich]

it is already configurable, you can set UCX_CUDA_COPY_DMABUF=n to disable dmabuf

[erezzarum]

I do not want to disable dmabuf.
My question is, why do we check the presence of efa_nv_peermem module, if we do not use it?
My assumption this validation assume that the presence of this module means the instance is capable of GPUDirect RDMA, while this is true, this is for legacy nvpeermem and not for dmabuf, so validating the presence of this module to decide if GPUDirect RDMA is supported is not the correct way.

[brminich]

I may not fully understand your question. UCX checks both presence of efa_nv_peermem and dmabuf support. If either of them is present/supported, UCX assumes GPUDirect RDMA can be used

[erezzarum]

I have tested on two different instances that supports GPUDirect RDMA dmabuf, p5.48xlarge and p5en.48xlarge, without loading the efa_nv_peermem module, UCX fallback to "software emulation", after loading the efa_nv_peermem, UCX was using cuda correctly.
I have tested libfabric and it works correct with dmabuf without having the efa_nv_peermem loaded, so something with UCX either not able to use dmabuf with EFA or this validation requires efa_nv_peermem always even though it uses dmabuf.

[erezzarum]

it looks like UCX was not able to use dmabuf, so it fallback to nv_peermem

[1761131060.014794] [nixl-benchmark-869cfdfbd9-pmt6p:29   :0]           ib_md.c:1222 UCX  DEBUG rdmap183s0: cuda GPUDirect RDMA is not detected by checking /sys/kernel/mm/memory_peers/nv_mem/version
[1761131060.014801] [nixl-benchmark-869cfdfbd9-pmt6p:29   :0]           ib_md.c:1222 UCX  DEBUG rdmap183s0: cuda GPUDirect RDMA is not detected by checking /sys/module/nvidia_peermem/version
[1761131060.014811] [nixl-benchmark-869cfdfbd9-pmt6p:29   :0]           ib_md.c:1222 UCX  DEBUG rdmap183s0: cuda GPUDirect RDMA is not detected by checking /sys/module/nv_peer_mem/version
[1761131060.014819] [nixl-benchmark-869cfdfbd9-pmt6p:29   :0]           ib_md.c:1222 UCX  DEBUG rdmap183s0: rocm GPUDirect RDMA is not detected by checking /dev/kfd
[1761131060.014832] [nixl-benchmark-869cfdfbd9-pmt6p:29   :0]           ib_md.c:1243 UCX  DEBUG rdmap183s0: ibv_reg_dmabuf_mr(fd=-1) returned Operation not supported, dmabuf is not supported
[1761131060.014836] [nixl-benchmark-869cfdfbd9-pmt6p:29   :0]           ib_md.c:1067 UCX  DEBUG rdmap183s0: md open by 'uct_ib_efa_md_ops' is successful
[1761131350.739469] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]            topo.c:710  UCX  DEBUG rdmap184s0: bdf_name 0000:b8:00.0 sys_dev 4
[1761131350.739495] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]       ib_device.c:537  UCX  DEBUG rdmap184s0: vendor_id 0x1d0f device_id 61345
[1761131350.739582] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]           ib_md.c:1222 UCX  DEBUG rdmap184s0: cuda GPUDirect RDMA is detected by checking /sys/module/efa_nv_peermem/version
[1761131350.739612] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]           async.c:247  UCX  DEBUG added async handler 0x55c51db8f320 [id=54 ref 1] ???() to hash
[1761131350.739636] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]           async.c:521  UCX  DEBUG listening to async event fd 54 events 0x1 mode thread_spinlock
[1761131350.739641] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]       ib_device.c:648  UCX  DEBUG initialized device 'rdmap184s0' (unspecified) with 1 ports
[1761131350.739652] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]           ib_md.c:1222 UCX  DEBUG rdmap184s0: rocm GPUDirect RDMA is not detected by checking /dev/kfd
[1761131350.739669] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]           ib_md.c:1243 UCX  DEBUG rdmap184s0: ibv_reg_dmabuf_mr(fd=-1) returned Operation not supported, dmabuf is not supported
[1761131350.739675] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]           ib_md.c:1067 UCX  DEBUG rdmap184s0: md open by 'uct_ib_efa_md_ops' is successful
[1761131350.739958] [nixl-benchmark-869cfdfbd9-pmt6p:44   :0]  rc_verbs_iface.c:550  UCX  DEBUG rdmap184s0: ibv_create_qp(RC) failed: Operation not supported

[brminich]

so what is the exact issue? looks like on this particular system dmabuf is not supported (can be various reasons for that)