prov/efa/shm: CMA error 14 when MPI ranks > 1000

[yuanjianz]

Describe the bug
I am running some HPC applications on AWS parallelcluster using hpc7a instances (192 cores per node). My applications crash when total MPI ranks > 1000 when MPI_Bcast. Turning FI_LOG_LEVEL=warning, the following messages are shown:

libfabric:54609:1754681986::core:core:cma_copy():58<warn> CMA error 14
libfabric:54609:1754681986::shm:ep_ctrl:smr_start_common():790<warn> error processing op
[hpc7a-2b-dy-hpc7a-192-1:54137] *** An error occurred in MPI_Bcast
[hpc7a-2b-dy-hpc7a-192-1:54137] *** reported by process [2390687745,0]
[hpc7a-2b-dy-hpc7a-192-1:54137] *** on communicator MPI COMMUNICATOR 25 GROUP FROM 21
[hpc7a-2b-dy-hpc7a-192-1:54137] *** MPI_ERR_INTERN: internal error
[hpc7a-2b-dy-hpc7a-192-1:54137] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[hpc7a-2b-dy-hpc7a-192-1:54137] ***    and potentially your MPI job)
[hpc7a-2b-dy-hpc7a-192-1:54104] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[hpc7a-2b-dy-hpc7a-192-1:54104] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

By disabling shm in mtl/ofi,

export FI_EFA_ENABLE_SHM_TRANSFER=0
export OMPI_MCA_mtl_ofi_provider_exclude=shm

the application run successfully with the following warnings:

libfabric:57215:1754682263::efa:ep_ctrl:efa_rdm_ep_post_handshake():610<warn> PKE entries exhausted.
libfabric:57215:1754682263::efa:ep_ctrl:efa_rdm_ep_post_handshake():610<warn> PKE entries exhausted.
...

To Reproduce
I guess MPI_Bcast at large scale with efa/shm could possibly trigger this error? I am not sure if this is application-specific.

Environment:
Amazon Linux 2023.6.20250317, hpc7a ec2 instance, openmpi 4.1.7, libfabric-aws/1.22.0amzn5.0

Additional context
I also make sure ptrace is set correctly by inserting

if [[ $( cat /proc/sys/kernel/yama/ptrace_scope ) == "0" ]]; then
        echo "PTrace Correct for EFA"
else
        echo "PTrace Override"
        sysctl -w kernel.yama.ptrace_scope=0
fi

as a part of my execution script and run mpirun ./execute.sh.

[shijin-aws]

Thanks for the detailed report.

We have CI running on 6 hpc7a (1152 ranks totally) and it looks osu_bcast passed

INFO     root:utils.py:72 Executing command: export PATH=/opt/amazon/openmpi/bin:$PATH;export FI_EFA_USE_DEVICE_RDMA=1; export LD_LIBRARY_PATH=/opt/amazon/efa/lib64;/opt/amazon/openmpi/bin/mpirun --wdir . -n 1152 --hostfile /fsx/PortaFiducia/hostfile --map-by ppr:192:node --timeout 1800 -x FI_EFA_USE_DEVICE_RDMA=1 -x LD_LIBRARY_PATH=/opt/amazon/efa/lib64 -x PATH  /fsx/PortaFiducia/build/workloads/omb/openmpi-v4.1.7-installer/install/libexec/osu-micro-benchmarks/mpi/collective/osu_bcast   2>&1 | tee /fsx/PortaFiducia/build/workloads/omb/openmpi-v4.1.7-installer/run/collective/osu_bcast/node6-ppn192.txt
INFO     root:utils.py:621 mpirun output:
Warning: Permanently added 'queue-hpc7a96xlarge-dy-hpc7a96xlarge-3,10.0.49.249' (ECDSA) to the list of known hosts.
Warning: Permanently added 'queue-hpc7a96xlarge-dy-hpc7a96xlarge-6,10.0.48.30' (ECDSA) to the list of known hosts.
Warning: Permanently added 'queue-hpc7a96xlarge-dy-hpc7a96xlarge-1,10.0.61.212' (ECDSA) to the list of known hosts.
Warning: Permanently added 'queue-hpc7a96xlarge-dy-hpc7a96xlarge-4,10.0.55.204' (ECDSA) to the list of known hosts.
Warning: Permanently added 'queue-hpc7a96xlarge-dy-hpc7a96xlarge-5,10.0.50.10' (ECDSA) to the list of known hosts.
Warning: Permanently added 'queue-hpc7a96xlarge-dy-hpc7a96xlarge-2,10.0.60.158' (ECDSA) to the list of known hosts.

# OSU MPI Broadcast Latency Test v7.0-lrbison3
# Size       Avg Latency(us)
1                       5.93
2                       5.65
4                       5.45
8                       5.88
16                      5.97
32                      5.98
64                      6.18
128                     6.50
256                     7.60
512                     7.81
1024                   10.00
2048                   13.84
4096                   21.54
8192                   38.48
16384                  72.76
32768                 148.68
65536                 305.73
131072                643.99
262144               1264.92
524288               4938.72
1048576              9788.62

It is on pcluster 3.9.1 + al2 though.

[shijin-aws]

@aingerson Do you have any insights what can cause that shm error?

libfabric:54609:1754681986::core:core:cma_copy():58<warn> CMA error 14
libfabric:54609:1754681986::shm:ep_ctrl:smr_start_common():790<warn> error processing op

[yuanjianz]

@shijin-aws thanks for the information. I just tried the osu_bcast, it also passed with the same configuration of my application (I just replaced the ./application with ./osu_bcast)

With 1152 cores across 6 nodes, I got the aforementioned errors for my application but passed the osu_bcast.

# OSU MPI Broadcast Latency Test v7.3
# Datatype: MPI_CHAR.
# Size       Avg Latency(us)
1                       4.59
2                       4.46
4                       4.18
8                       4.70
16                      4.87
32                      4.47
64                      5.10
128                     5.37
256                     5.91
512                     6.29
1024                    7.46
2048                   10.10
4096                   16.62
8192                   31.20
16384                  49.58
32768                  99.02
65536                 207.30
131072                455.58
262144                882.14
524288               5197.53
1048576             10109.36

My slurm script sets some other env variables

# Recommended by AWS support
export OMPI_MCA_btl=vader,tcp,self
export OMPI_MCA_btl_tcp_if_exclude="lo,docker0,virbr0"
export OMPI_MCA_btl_if_exclude="lo,docker0,virbr0"

mpiexec -np ${SLURM_NTASKS} --map-by core --bind-to core ./execute.sh > ${log}

#!/bin/bash
# execute.sh

# Recommended by AWS support
if [[ $( cat /proc/sys/kernel/yama/ptrace_scope ) == "0" ]]; then
        echo "PTrace Correct for EFA"
else
        echo "PTrace Override"
        sysctl -w kernel.yama.ptrace_scope=0
fi

./application

[shijin-aws]

@yuanjianz

export OMPI_MCA_btl=vader,tcp,self

It is weird that you are using vader/tcp for btl while OFI for MTL, I would drop that env first

[shijin-aws]

For that efa warning

libfabric:57215:1754682263::efa:ep_ctrl:efa_rdm_ep_post_handshake():610<warn> PKE entries exhausted.
libfabric:57215:1754682263::efa:ep_ctrl:efa_rdm_ep_post_handshake():610<warn> PKE entries exhausted.

It is actually harmless. It is temporary resource unavailable and efa will retry later. I moved this logging to debug level in Libfabric 2.x versions f92bf93, 1.22.0amzn5.0 hasn't have that yet.

[shijin-aws]

But generally shm CMA not working and disabling shm will hurt intra-node performance, I would consult shm people @aingerson to get a root cause

[yuanjianz]

@shijin-aws
Unfortunately, my application will crash later (likely during MPI_Put somewhere but I never confirmed) if I do not exclude OFI from BTL. But it seems like a separate issue.
I tried comment out the export OMPI_MCA_btl=vader,tcp,self but it still crashes here in the MPI_Bcast.

[aingerson]

I don't think it should be relevant that you're setting btl also. It should be ignored. You should be using either btl or mtl which is determined by your pml setting (cm->mtl, ob1->btl). So first can you clarify which pml you are setting?

I'm not sure about the error. That error means either the remote iov address or local iov address is incorrect which I've only seen if the app/caller of OFI uses the wrong address, usually in RMA when passing in a 0 based offset when it expects a virtual address.

Could you add the following debug print in your libfabric source to see what addresses are getting passed in to your CMA copy before the crash?

diff --git a/include/ofi_cma.h b/include/ofi_cma.h
index ade1b74c21..7e236ef744 100644
--- a/include/ofi_cma.h
+++ b/include/ofi_cma.h
@@ -57,6 +57,10 @@ static inline int cma_copy(struct iovec *local, unsigned long local_cnt,
                if (ret < 0) {
                        FI_WARN(&core_prov, FI_LOG_CORE,
                                "CMA error %d\n", errno);
+                       FI_WARN(&core_prov, FI_LOG_CORE,
+                               "local: %p, local_cnt: %lu, local[0]: %p "
+                               "--- remote: %p, remote_cnt: %lu, remote[0]: %p\n",
+                               local, local_cnt, local[0], remote, remote_cnt, remote[0]);
                        return -FI_EIO;
                }

[shijin-aws]

The source code of 1.22.0amzn5.0 is in https://github.com/aws/libfabric/commits/1.22.0amzn5.0/

Do you know how to build from source ? @yuanjianz

[yuanjianz]

@shijin-aws not really. I would appreciate it if you could provide some guidance on how to build it, maybe some needed flags?

I guess since I am doing all these in pcluster I need to build another libfabric in my shared fsx instead of keep using the libfabric in /opt. Do I also need to specify the libfabric path to OpenMPI?

[shijin-aws]

I guess since I am doing all these in pcluster I need to build another libfabric in my shared fsx instead of keep using the libfabric in /opt. Do I also need to specify the libfabric path to OpenMPI?

Yes, you can clone the source code to your fsx via

git clone -b 1.22.0amzn5.0 https://github.com/aws/libfabric

And follow https://github.com/ofiwg/libfabric/blob/main/prov/efa/docs/building.md to build it

Do I also need to specify the libfabric path to OpenMPI?

You just need to specify your LD_LIBRARY_PATH to this customized libfabric path via

mpirun -x LD_LIBRARY_PATH=<your_libfabric_install_path>/lib:$LD_LIBRARY_PATH ...

[shijin-aws]

Actually the

./configure --enable-efa=<path to rdma-core install> --prefix=$PWD/install

in https://github.com/ofiwg/libfabric/blob/main/prov/efa/docs/building.md can be simplified as

./configure --enable-efa --prefix=<your_libfabric_install_path>

You are using pcluster so the rdma-core should be installed in /usr already

[yuanjianz]

@shijin-aws thanks for your help.

@aingerson I guess pml is cm since disabling shm from mtl/ofi/efa bypass this error.

libfabric:27290:1755716986::core:core:cma_copy():58<warn> CMA error 14
libfabric:27290:1755716986::core:core:cma_copy():60<warn> local: 0x145dffb6bb28, local_cnt: 1, local[0]: 0xaa9d430 --- remote: 0x2ab250, remote_cnt: 22397348428184, remote[0]: 0x1
libfabric:27290:1755716986::shm:ep_ctrl:smr_start_common():790<warn> error processing op

[aingerson]

I think you're probably running mtl but I don't think that's quite the logic. You said the following bypasses the error:

export FI_EFA_ENABLE_SHM_TRANSFER=0
export OMPI_MCA_mtl_ofi_provider_exclude=shm

but I think the second line isn't doing anything because shm gets picked up internally by efa. shm is not a valid ompi mtl provider so it will never get returned on that level. I think the important line is the first one because that disables shm on the efa level.

Thanks for providing the debug information! Sorry I printed out the wrong iov address 🤦‍♀️ can you try again with the following fixed patch? It also includes the op type so I can take a closer look at the failing path. Thank you!

diff --git a/include/ofi_cma.h b/include/ofi_cma.h
index ade1b74c21..282c395f98 100644
--- a/include/ofi_cma.h
+++ b/include/ofi_cma.h
@@ -57,6 +57,11 @@ static inline int cma_copy(struct iovec *local, unsigned long local_cnt,
                if (ret < 0) {
                        FI_WARN(&core_prov, FI_LOG_CORE,
                                "CMA error %d\n", errno);
+                       FI_WARN(&core_prov, FI_LOG_CORE,
+                               "local: %p, local_cnt: %lu, local[0].iov_base: %p "
+                               "--- remote: %p, remote_cnt: %lu, remote[0].iov_base: %p\n",
+                               local, local_cnt, local[0].iov_base,
+                               remote, remote_cnt, remote[0].iov_base);
                        return -FI_EIO;
                }
 
diff --git a/prov/shm/src/smr_progress.c b/prov/shm/src/smr_progress.c
index cafb375639..aa8441616f 100644
--- a/prov/shm/src/smr_progress.c
+++ b/prov/shm/src/smr_progress.c
@@ -766,7 +766,7 @@ static int smr_start_common(struct smr_ep *ep, struct smr_cmd *cmd,
                                             cmd->msg.hdr.op_flags);
                if (err) {
                        FI_WARN(&smr_prov, FI_LOG_EP_CTRL,
-                               "error processing op\n");
+                               "error processing op %d\n", cmd->msg.hdr.op);
                        ret = smr_write_err_comp(ep->util_ep.rx_cq,
                                                 rx_entry->context,
                                                 comp_flags, rx_entry->tag,