prov/efa: Potential atomicity issue with ofi_copy_to_iov() for NCCL_PROTO=LL128 protocol support

[zhou-yukun]

In the context of NCCL_PROTO=LL128, which requires atomicity guarantees for received data within 128-byte boundaries, I've identified a potential issue in the current implementation.

In the function efa_rdm_pke_copy_payload_to_ope(), when handling non-CUDA and non-HMEM memory types (specifically system memory), the code uses ofi_copy_to_iov for data copying:

bytes_copied = ofi_copy_to_iov(ope->iov, ope->iov_count, segment_offset + ep->msg_prefix_size, pke->payload, pke->payload_size);

The ofi_copy_to_iov function ultimately calls memcpy, which may not guarantee atomicity for 128-byte data transfers. This could potentially violate the atomicity requirements of the LL128 protocol.

Questions:

1. Is this a genuine concern for LL128 protocol compliance?
2. Should system memory copies also have atomicity guarantees for 128-byte boundaries?
3. If this is indeed an issue, what would be the recommended approach to ensure atomicity for system memory copies in LL128 scenarios?

Thank you for your attention to this issue.

[shijin-aws]

@zhou-yukun Thanks for reporting the issue. Today ofi nccl plugin (where libfabric is accessed by NCCL) is using FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES/FI_OPT_EFA_WRITE_IN_ORDER_ALIGNED_128_BYTES to enforce 128 bytes atomicity from libfabric. Today EFA provider only apply this option to cuda memory transfer (by copying the data from internal bounce buffer to the dest cuda memory via a local rdma read which guarnateed that 128 bytes atomicity), but according to

libfabric/man/fi_efa.7.md

Lines 154 to 160 in 5b39be8

	*FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES - bool*
	: This option only applies to the fi_setopt() call.
	It is used to force the endpoint to use in-order send/recv operation for each 128 bytes
	aligned block. Enabling the option will guarantee data inside each 128 bytes
	aligned block being sent and received in order, it will also guarantee data
	to be delivered to the receive buffer only once. If endpoint is not able to
	support this feature, it will return -FI_EOPNOTSUPP for the call to fi_setopt().

, this option shouldn't be restricted to cuda. So we need to fix it for other memory types

[zhou-yukun]

您好!您的邮件已收到!我查阅后会尽快回复您!

[zhou-yukun]

@shijin-aws Thank you for your reply. I apologize for the delayed response due to recent personal matters.

I understand that libfabric currently ensures 128-byte atomicity for CUDA memory transfers through certain parameter settings, but such guarantees are not yet available for other memory types.

However, I noticed that when running an NCCL test with NCCL_PROTO=LL128 specified, the memory operations seem to follow a specific sequence: the data is first copied to the CPU side (host-type memory), and then transferred to the GPU for processing. Therefore, the step that requires 128-byte atomicity is actually the RDMA receive operation copying data to the CPU side—which essentially involves host-type memory. Do you have any suggestions on how to address this issue?

@zhou-yukun Thanks for reporting the issue. Today ofi nccl plugin (where libfabric is accessed by NCCL) is using FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES/FI_OPT_EFA_WRITE_IN_ORDER_ALIGNED_128_BYTES to enforce 128 bytes atomicity from libfabric. Today EFA provider only apply this option to cuda memory transfer (by copying the data from internal bounce buffer to the dest cuda memory via a local rdma read which guarnateed that 128 bytes atomicity), but according to

libfabric/man/fi_efa.7.md

Lines 154 to 160 in 5b39be8

FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES - bool
: This option only applies to the fi_setopt() call.
It is used to force the endpoint to use in-order send/recv operation for each 128 bytes
aligned block. Enabling the option will guarantee data inside each 128 bytes
aligned block being sent and received in order, it will also guarantee data
to be delivered to the receive buffer only once. If endpoint is not able to
support this feature, it will return -FI_EOPNOTSUPP for the call to fi_setopt().
, this option shouldn't be restricted to cuda. So we need to fix it for other memory types

[shijin-aws]

However, I noticed that when running an NCCL test with NCCL_PROTO=LL128 specified, the memory operations seem to follow a specific sequence: the data is first copied to the CPU side (host-type memory), and then transferred to the GPU for processing. Therefore, the step that requires 128-byte atomicity is actually the RDMA receive operation copying data to the CPU side—which essentially involves host-type memory. Do you have any suggestions on how to address this issue?

It depends on which code path of EFA provider is. For the efa fabric of EFA provider (where the efa_rdm_pke_copy_payload_to_ope will happen in your original question), it has internal host memory bounce buffer to receive data delivered by the NIC, and copy them to the destination buffer. The 128-byte atomicity should happen on the copy operation from the bounce buffer to the destination buffer, no matter what memory type of the dest buffer is.

Currently Libfabric handles this atomicity when copying data from host to cuda by performing a local RDMA read which supports 128 bytes aligned delivery. However, this mechanism is not applied to non-cuda destination buffer, which is a bug.
The solution will be enforcing the 128 byte RDMA read copy method for all dest memory type.

There is another fabric called efa-direct, which offload the user posted rx buffer to efa device directly. No copy will happen in this code path, libfabric offloads the 128 byte atomicity requirement to device completely, i.e the RDMA receive operation

[zhou-yukun]

您好!您的邮件已收到!我查阅后会尽快回复您!

[staryxchen]

Currently Libfabric handles this atomicity when copying data from host to cuda by performing a local RDMA read which supports 128 bytes aligned delivery. However, this mechanism is not applied to non-cuda destination buffer, which is a bug.
The solution will be enforcing the 128 byte RDMA read copy method for all dest memory type.

Hi @shijin-aws
I want to know if the RDMA read-copy method can achieve atomic transfer of 128-byte data from non-CUDA/CUDA memory to non-CUDA memory? If so, is it better to unconditionally replace ofi_copy_to_iov with efa_rdm_pke_copy_payload_to_cuda?

[shijin-aws]

Currently Libfabric handles this atomicity when copying data from host to cuda by performing a local RDMA read which supports 128 bytes aligned delivery. However, this mechanism is not applied to non-cuda destination buffer, which is a bug.
The solution will be enforcing the 128 byte RDMA read copy method for all dest memory type.

Hi @shijin-aws I want to know if the RDMA read-copy method can achieve atomic transfer of 128-byte data from non-CUDA/CUDA memory to non-CUDA memory? If so, is it better to unconditionally replace ofi_copy_to_iov with efa_rdm_pke_copy_payload_to_cuda?

Yes, rdma read copy have that atomicity for all memory type, that is our plan to fix it . basically in that way, we will make efa_rdm_pke_copy_payload_to_ope to use rdma when FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES is true

[shijin-aws]

We should post a PR to fix it soon, thanks for your patience!