libfabric verbs provider not working on Windows 11 (fi_info -p verbs → -120 “no message available” / no RDMA devices found)

[ficipo]

I’m trying to use the libfabric verbs provider on Windows 11 with Mellanox ConnectX-6 NIC for RDMA communication, but fi_info -p verbs and the fi_pingpong test do not work as expected. fi_info.exe doesn’t even show the devices which use verbs provider, but with -l argument I can see that verbs is listed (version: 203.10). Running the same version of libfabric on Ubuntu 22.04 with identical Mellanox hardware detects verbs correctly, so this doesn’t appear to be a hardware issue.

I'm getting ret=-120 "no message available".
Here is the full log output with FI_LOG_LEVEL=info set:

fi_info -p verbs

Click to expand full log

C:\Users\Admin\Desktop\Test\libfabric-2.3.1\x64\Release-v142>fi_info.exe -p verbs
libfabric:36824:1762953367::core:core:fi_param_get():373 variable perf_cntr=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable hook=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable hmem=
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_CUDA not supported
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_ROCR not supported
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_ZE not supported
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_NEURON not supported
libfabric:36824:1762953367::core:core:ofi_hmem_init():667 Hmem iface FI_HMEM_SYNAPSEAI not supported
libfabric:36824:1762953367::core:core:fi_param_get():373 variable hmem_disable_p2p=
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor uffd
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor memhooks
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor cuda
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor cuda_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor rocr
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor rocr_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor xpmem
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor ze
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor ze_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor import
libfabric:36824:1762953367::core:mr:ofi_monitors_init():240 Initializing memory monitor kdreg2
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_cache_max_size=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_cache_max_count=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_cache_monitor=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_cuda_cache_monitor_enabled=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_rocr_cache_monitor_enabled=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable mr_ze_cache_monitor_enabled=
libfabric:36824:1762953367::core:mr:ofi_default_cache_size():67 default cache size=1048576
libfabric:36824:1762953367::core:mr:ofi_monitors_init():317 Default memory monitor is: disabled
libfabric:36824:1762953367::core:core:fi_param_get():373 variable provider=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable fork_unsafe=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable universe_size=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable av_remove_cleanup=
libfabric:36824:1762953367::core:core:fi_param_get():373 variable offload_coll_provider=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable enable_passthru=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable buffer_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable tx_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable rx_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable msg_tx_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable msg_rx_size=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable cm_progress_interval=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable cq_eq_fairness=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable data_auto_progress=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable use_rndv_write=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable def_wait_obj=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable def_tcp_wait_obj=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable detect_hmem_iface=
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable rescan=
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: ofi_rxm (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: verbs (203.10)
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable spin_count=
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable retry=
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable max_peers=
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable max_unacked=
libfabric:36824:1762953367::ofi_rxd:core:fi_param_get():373 variable rescan=
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: ofi_rxd (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: udp (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: sockets (203.10)
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable prov_name=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable port_high_range=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable port_low_range=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable tx_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable rx_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable max_inject=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable max_saved=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable max_saved_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable max_rx_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable nodelay=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable staging_sbuf_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable prefetch_rbuf_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable zerocopy_size=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable trace_msg=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable disable_auto_progress=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable io_uring=
libfabric:36824:1762953367::tcp:core:fi_param_get():373 variable firewall_addr=
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: tcp (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: ofi_hook_perf (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: ofi_hook_noop (203.10)
libfabric:36824:1762953367::core:core:ofi_register_provider():528 registering provider: off_coll (203.10)
libfabric:36824:1762953367::verbs:core:nd_startup():744 nd_startup: starting initialization
libfabric:36824:1762953367::verbs:core:nd_startup():750 nd_startup: WSAStartup complete
libfabric:36824:1762953367::verbs:core:nd_create_factory():436 nd_create_factory: provider 7C4DB90D-6125-43B0-9267-2617E62DAAF2 path: C:\WINDOWS\system32\ch_nd.dll
libfabric:36824:1762953367::verbs:core:nd_create_factory():436 nd_create_factory: provider 134043E9-5D77-47BE-AFA9-5BC722D6F49A path: C:\WINDOWS\system32\mlx5nd.dll
libfabric:36824:1762953367::verbs:core:nd_create_factory():436 nd_create_factory: provider B491F6A6-F964-4111-8DCC-89174580C020 path: C:\WINDOWS\system32\mlx5nd.dll
libfabric:36824:1762953367::verbs:fabric:vrb_init():776 failed in OS specific device initialization
libfabric:36824:1762953367::core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable use_srx=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable tx_size=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable rx_size=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable tx_iov_limit=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable rx_iov_limit=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable inline_size=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable min_rnr_timer=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable use_odp=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable prefer_xrc=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable xrcd_filename=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable cqread_bunch_size=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable gid_idx=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable device_name=
libfabric:36824:1762953367:ofi_rxm:verbs:core:vrb_read_params():739 dmabuf support is disabled
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable iface=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable dgram_use_name_server=
libfabric:36824:1762953367:ofi_rxm:verbs:core:fi_param_get():373 variable dgram_name_server_port=
libfabric:36824:1762953367:ofi_rxm:verbs:fabric:ibv_get_device_list():50
libfabric:36824:1762953367:ofi_rxm:verbs:fabric:vrb_init_info():1422 no RDMA devices found
libfabric:36824:1762953367:ofi_rxm:core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1265 Provider ofi_rxm is excluded
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1276 Need core provider, skipping ofi_rxd
libfabric:36824:1762953367::ofi_rxm:core:ofi_check_fabric_attr():470 Requesting provider verbs, skipping tcp
libfabric:36824:1762953367::ofi_rxm:core:ofi_check_fabric_attr():470 Requesting provider verbs, skipping tcp
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable use_srx=
libfabric:36824:1762953367:ofi_rxm:core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1265 Provider ofi_rxm is excluded
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1276 Need core provider, skipping ofi_rxd
libfabric:36824:1762953367::ofi_rxm:core:fi_param_get():373 variable use_srx=
libfabric:36824:1762953367:ofi_rxm:core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1265 Provider ofi_rxm is excluded
libfabric:36824:1762953367:ofi_rxm:core:core:ofi_layering_ok():1276 Need core provider, skipping ofi_rxd
libfabric:36824:1762953367::core:core:fi_getinfo():1386 fi_getinfo: provider ofi_rxm returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxd:core:core:fi_getinfo():1386 fi_getinfo: provider verbs returned -120 (no message available)
libfabric:36824:1762953367:ofi_rxd:core:core:ofi_layering_ok():1276 Need core provider, skipping ofi_rxm
libfabric:36824:1762953367:ofi_rxd:core:core:ofi_layering_ok():1265 Provider ofi_rxd is excluded
libfabric:36824:1762953367::core:core:fi_getinfo():1386 fi_getinfo: provider ofi_rxd returned -120 (no message available)
fi_getinfo: -120 (no message available)
libfabric:36824:1762953367::verbs:core:nd_shutdown():764 nd_shutdown: shutdown WSA
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor uffd
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor memhooks
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor cuda
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor cuda_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor rocr
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor rocr_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor xpmem
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor ze
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor ze_ipc
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor import
libfabric:36824:1762953367::core:mr:ofi_monitors_cleanup():340 Cleaning up memory monitor kdreg2

Here is the Get-NetAdapterRdma output form Winodws PowerShell:
Name InterfaceDescription Enabled Operational PFC ETS

---

Ethernet Mellanox ConnectX-6 Dx Adapter #3 True False False False
vEthernet (Default Hyper-V Virtual Ethernet Adapter False False NA NA
vEthernet (WSL (H Hyper-V Virtual Ethernet Adapter #2 False False NA NA
Ethernet 2 Mellanox ConnectX-6 Dx Adapter #4 True False False False

System Configuration:
-OS: Windows 11 Pro(24H2)
-Hardware: Mellanox ConnectX-6 Dx 100 GbE
-Driver: WinOF-2 version 25.7.50000 Windows Client 11, https://network.nvidia.com/products/adapter-software/ethernet/windows/winof-2/
-libfabric version: 2.3.1 (built from source using Visual Studio 2022, Release-v142 x64),extracted header files from downloaded NetworkDirect_DDK.zip:\NetDirect\include\ into include\windows
-RoCE: Enabled in mlxconfig (ROCE_V2=1, ROCE_ENABLE=2)