prov/lnx: shm+cxi:lnx only shm is working

[andreaa93]

Describe the bug
Between libfabric 2.0 and 2.2.0 we observed that the variable OMPI_MCA_opal_common_ofi_provider_include shm+cxi:lnx works only for shm and yields degraded performances between different nodes

To Reproduce
our openmpi module contain the following

setenv          FI_CXI_RX_MATCH_MODE software
setenv          FI_SHM_USE_XPMEM 1
setenv          FI_LNX_PROV_LINKS shm+cxi
setenv          PALS_PMI pmix
setenv          PALS_CPU_BIND none
setenv          OMPI_MCA_mtl ofi
setenv          OMPI_MCA_pml ^ucx
setenv          OMPI_MCA_opal_common_ofi_provider_include shm+cxi:lnx
setenv          PRTE_MCA_ras_base_launch_orted_on_hn 1

mpirun -np 2 opt/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw --validation

Expected behavior
Bandwidth performance should be good between two nodes, as observed for libfabric 2.0

Output

Libfabric 2.2.0 performances

# OSU MPI Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)        Validation
2097152              4435.40              Pass

Libfabric 2.0 performances

# OSU MPI Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)        Validation
2097152             22587.12              Pass

Environment:
Linux node 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Mar 14 14:20:09 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Additional context
NIC is Slingshot 11 on HPE cluster

[j-xiong]

If only shm is working, how does communication between different nodes happen?

[amirshehataornl]

The rework we did with lnx requires a different format for OMPI_MCA_opal_common_ofi_provider_include
Can you try setting:

OMPI_MCA_opal_common_ofi_provider_include lnx

Then there is an environment variable that you set to define the exact link. Here is the relevant part for the man page:

*FI_LNX_PROV_LINKS*
: This environment variable is used to specify which providers to link. This
  must be set in order for the LNX provider to return a list of fi_info
  blocks in the fi_getinfo() call. The format which must be used is:
       link ::= group ['|' group]*
       group ::= provider ['+' provider [':' provider_list]]
       provider_list ::= provider [',' provider]*
       provider ::= [a-zA-Z0-9_]+
  Examples:
     - shm+cxi:cxi0
     - shm+cxi:cxi0,cxi1
     - shm+cxi:cxi0|shm+cxi:cxi1
     - shm+cxi
  On a system with four cxi domains, the last example is equivalent to:
     - shm+cxi:cxi0,cxi1,cxi2,cxi3

Can you try these config changes and let me know how the system behaves.

What you're likely seeing is open MPI is failing to initialize libfabric because it can't find the provider. It falls back to using its own internal shared memory implementation and probably just tcp

[andreaa93]

After some tests I found with my colleagues (@francescofinelli94 @Fruchix et al.) the solution.

Changing the syntax as you indicated was necessary but not sufficient.

We had to recompile on our machine the libfabric 2.2 with libcxi using shs package at version 12.0.1, instead of version 11.1.0 as we had before on our system.

So the two things together made it work.

Did you expect your suggestion to work with libfabric 2.2 and shs 11.1.0?

Thank you very much for your prompt answer, cheers
Andrea

[amirshehataornl]

Yes I was expecting it to work regardless of the shs version. It's not clear to me what the incompatibility is.

[amirshehataornl]

Now that I think about it there could be an incompatibility between the CXI provider version and shs 11.1.0? CXI provider calls into the cxil library which I think would be part of the shs package. There could be some compatibility requirements between these two components. HPE folks would be able to verify for sure.

[andreaa93]

Hello @PourroyJean, do you think this check could be carried out by you guys?

[andreaa93]

FYI @Giuncan