fabtests on ConnectX-7 with "verbs;ofi_rxm" provider fails to run

[abcdabcd987]

Hi, I'm trying to run fabtests on CX-7, but it hangs. Here's my command:

./fi_rma_bw -o writedata -E -S all -W 64 -p "verbs;ofi_rxm" -d mlx5_0

Here's the FI_LOG_INFO=debug log: rc.txt

And for reference, here's the output of fi_info -p "verbs;ofi_rxm" -d mlx5_0 -v: fi_info.yaml

[ooststep]

In your log, I see the following error

libfabric:1923604:1758154709::ofi_rxm:cq:rxm_handle_error():789<warn> fi_eq_readerr: err: Connection refused (111), prov_err: Unknown error -28 (-28)
libfabric:1923604:1758154709::ofi_rxm:ep_ctrl:rxm_process_reject():537<info> Processing reject for handle: 0x5736981b8ba0
libfabric:1923604:1758154709::verbs:domain:vrb_ep_close():684<info> EP 0x573697f5e990 is being closed

-28 is FI_ENOSPC. Is there any indication of an error in dmesg or other system logs when this occurs?

Can you also provide the client-side log?

[abcdabcd987]

Here's the client log: client.txt

For dmesg, all I have is this (not sure if relevant):

[160984.636303] audit: type=1400 audit(1758082780.652:5441): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="soperator-cluster-soperator-ns-soperator" pid=6508 comm="security-profil"

And as a sanity check, I confirm that rdma-core perftest can run:

ib_write_bw -c RC -a -n 1024 --report_gbits --write_with_imm --inline_size 0 --ib-dev=mlx5_0 --use_cuda=4

[j-xiong]

The prov_err here in the eq entry is the status field of the lower-level cm event. 28 just means the reason is "consumer defined", it is not a value for errno.

What matters is that the connection request is refused by the peer. For anyone that is interested, the logic to reject the connection is in the function rxm_process_connreq (prov/rxm/src/rxm_conn.c).

Do you have IP address assigned to the cx-7 interface?

[abcdabcd987]

No IP is set for mlx5_* interfaces

[j-xiong]

Which version of libfabric are you using? Until very recently (v2.3.0), rxm/verbs requires IP address be assigned to the RDMA NIC to work properly.

[abcdabcd987]

Oh. Good to know. I didn't know that IP is required. But also, I'm using v2.3.0, built from source (you can see that from the log).