Question on verbs;ofi_rxm rendezvous serialization due to duplex QP

[dragosargint]

Hi,

I'm running IMB-MPI[1] benchmarks using Open MPI[2] with the verbs;ofi_rxm provider in Libfabric. While analysing performance, I encountered a serialisation issue affecting certain collective operations, particularly with large messages.

Here's the scenario I'm observing:

Consider two hosts, h1 and h2, each performing MPI_Send operations in parallel. From what I understand, for large messages, ofi_rxm uses a rendezvous protocol. For example, on h1, the following sequence of verbs operations occurs:

ibv_send()  => rndv_ctrl_req  
ibv_recv()  => rndv_ctrl_write  
ibv_write()  
ibv_send()  => rndv_ctrl_done

Because QPs are used as duplex channels, if the rndv_ctrl_req from h2 arrives at h1 while its ibv_write() is in progress, the response (rndv_ctrl_write) is delayed. This causes h2 to stall waiting for the signal to proceed with its own ibv_write(), serialising what should be parallel transfers.

One potential solution I’m considering is using two QPs per connection, where each direction uses its own QP (i.e. treating QPs as simplex channels)
Is there a way to configure RXM (or Libfabric more generally) to use multiple QPs per connection in this manner?

Thanks,
Dragos

[1] https://github.com/intel/mpi-benchmarks
[2] https://github.com/open-mpi/ompi

[dragosargint]

I opened an issue instead.