prov/ucx: Segmentation fault on fi_cq_read

[AOA-Mohammed]

After creating endpoints and exchanging addresses, registering memory for FI_REMOTE_READ and exchanging raw MR keys,

The application trying to read crashes with the following backtrace.

Using libfabric version 2.3 on MLX NDR200 NICs

backtrace

#0  0x00007ff11ca4854c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007ff11c9fbd46 in raise () from /lib64/libc.so.6
#2  0x00007ff11c9cf7f3 in abort () from /lib64/libc.so.6
#3  0x00007ff11eb889d1 in ucs_fatal_error_message (file=0x243b8b <error: Cannot access memory at address 0x243b8b>, line=2382514, function=0x6 <error: Cannot access memory at address 0x6>, message_buf=0x7ff11ca4854c <__pthread_kill_implementation+284> "\211\305\367\335=") at debug/assert.c:38
#4  0x00007ff11eb8e12c in ucs_log_default_handler (file=0x243b8b <error: Cannot access memory at address 0x243b8b>, line=2382514, function=0x6 <error: Cannot access memory at address 0x6>, level=480544076, comp_conf=0x7ff0986055e0, format=0x12 <error: Cannot access memory at address 0x12>, ap=0x7ff098607d00) at debug/log.c:317
#5  0x00007ff11eb8d15c in ucs_log_dispatch (file=0x243b8b <error: Cannot access memory at address 0x243b8b>, line=2382514, function=0x6 <error: Cannot access memory at address 0x6>, level=480544076, comp_conf=0x7ff0986055e0, format=0x12 <error: Cannot access memory at address 0x12>) at debug/log.c:381
#6  0x00007ff015ca7ea0 in uct_ib_mlx5_completion_with_err (iface=0x243b8b, ecqe=0x245ab2, txwq=0x6, log_level=480544076) at mlx5/ib_mlx5_log.c:171
#7  0x00007ff015cd9e61 in uct_rc_mlx5_iface_handle_failure (ib_iface=0x243b8b, arg=0x245ab2, ep_status=6) at rc/accel/rc_mlx5_iface.c:299
#8  0x00007ff015caa5b2 in uct_ib_mlx5_check_completion_with_err (iface=0x243b8b, cq=0x245ab2, cqe=0x6) at mlx5/ib_mlx5.c:496
#9  0x00007ff015ca8cab in uct_ib_mlx5_check_completion (iface=0x243b8b, cq=0x245ab2, cqe=0x6, flags=480544076) at mlx5/ib_mlx5.c:477
#10 0x00007ff015cd3445 in uct_ib_mlx5_poll_cq (iface=<optimized out>, cq=0x62d000b98bf8, poll_flags=<optimized out>, check_cqe_cb=<optimized out>) at /apps/GPP/UCX/SRC/ucx-1.15.0/src/uct/ib/mlx5/ib_mlx5.inl:146
#11 uct_rc_mlx5_iface_poll_tx (iface=<optimized out>, poll_flags=<optimized out>) at rc/accel/rc_mlx5_iface.c:149
#12 uct_rc_mlx5_iface_progress (arg=<optimized out>, flags=<optimized out>) at rc/accel/rc_mlx5_iface.c:189
#13 uct_rc_mlx5_iface_progress_cyclic (arg=0x243b8b) at rc/accel/rc_mlx5_iface.c:194
#14 0x00007ff11ecc7158 in ucs_callbackq_dispatch (cbq=0x6) at /apps/GPP/UCX/SRC/ucx-1.15.0/src/ucs/datastruct/callbackq.h:211
#15 uct_worker_progress (worker=<optimized out>) at /apps/GPP/UCX/SRC/ucx-1.15.0/src/uct/api/uct.h:2777
#16 ucp_worker_progress (worker=0x243b8b) at core/ucp_worker.c:2889
#17 0x00007ff11f1757c0 in ucx_ep_progress () from /gpfs/scratch/ehpc01/de340/maestro/sep09/install/lib/libfabric.so.1
#18 0x00007ff11ef960b4 in ofi_cq_progress () from /gpfs/scratch/ehpc01/de340/maestro/sep09/install/lib/libfabric.so.1
#19 0x00007ff11ef945f5 in ofi_cq_readfrom () from /gpfs/scratch/ehpc01/de340/maestro/sep09/install/lib/libfabric.so.1
#20 0x00007ff11f44c5ba in fi_cq_read (cq=0x243b8b, buf=0x245ab2, count=1) at /gpfs/scratch/ehpc01/de340/maestro/sep09/install/include/rdma/fi_eq.h:400
#21 mstro_ofi__check_and_handle_cq (ep=0x620000000098, incoming_msg_handler=<optimized out>) at ofi.c:2917

[j-xiong]

Some info in the trace looks suspicious:

#5 ... file = 0x243b8b ...
#6 ... iface = 0x243b8b ...
#7 ... ib_iface = 0x243b8b ...
#13 ... arg=0x243b8b ...
#16 ... worker=0x243b8b ...
#20 ... cq=0x243b8b...

It's very unlikely that the same address can be interpreted as so many different object types.

Could you provider a reproducer?

[AOA-Mohammed]

Thanks, I attached here a small reproducer of the code. In this reproducer, I see a similar seg fault when calling fi_send/recv when trying to do ping pong.

Build:
make MPICC=mpiicx ENABLE_ASAN=1

run:
./run.sh

ucx_reproducer.tar.gz

[AOA-Mohammed]

@j-xiong Any insights on this issue? Were you able to reproduce it?

[j-xiong]

I could not reproduce. I had to use mpirun to run the test since there is no ddt on my system. With the binary built with address sanitizer enabled, I saw warnings like this:
You are trying to dlopen a libigc.so.1 shared library with RTLD_DEEPBIND flag which is incompatible with sanitizer runtime (see https://github.com/google/sanitizers/issues/611 for details). If you want to run libigc.so.1 library under sanitizers please remove RTLD_DEEPBIND from dlopen flags.
Binary built with address sanitizer disabled ran through the end.

[AOA-Mohammed]

@j-xiong Thanks a lot for your feedback. I got it to work with UCX 1.17 instead. Now, I have the reproducer code working; however, I notice that sometimes, when I increase the number of messages in the pingpong that this reproducer code is doing (around 1M messages), the code still fails with:

Are there any tuning environment variables that could be used in this case?

1: [gs13r3b53:2064199:0:2064199]    rc_iface.h:600  Assertion `iface->tx.in_pending || ucs_arbiter_group_is_empty(arb_group)' failed
1: ==== backtrace (tid:2064199) ====
1:  0  /apps/GPP/UCX/1.17.0/INTEL/lib/libucs.so.0(ucs_handle_error+0x3f4) [0x7f2aa0818c94]
1:  1  /apps/GPP/UCX/1.17.0/INTEL/lib/libucs.so.0(ucs_fatal_error_message+0xec) [0x7f2aa081612c]
1:  2  /apps/GPP/UCX/1.17.0/INTEL/lib/libucs.so.0(ucs_fatal_error_format+0x103) [0x7f2aa0816033]
1:  3  /apps/GPP/UCX/1.17.0/INTEL/lib/ucx/libuct_ib.so.0(uct_rc_mlx5_base_ep_am_bcopy+0x186) [0x7f2a161f7896]
1:  4  /apps/GPP/UCX/1.17.0/INTEL/lib/libucp.so.0(+0xe8187) [0x7f2aa09b0187]
1:  5  /apps/GPP/UCX/1.17.0/INTEL/lib/libucp.so.0(+0xe823d) [0x7f2aa09b023d]
1:  6  /apps/GPP/UCX/1.17.0/INTEL/lib/libucp.so.0(ucp_tag_send_nbx+0x8dc) [0x7f2aa09d0b8c]
1:  7  /apps/GPP/UCX/1.17.0/INTEL/lib/libucp.so.0(ucp_tag_send_nb+0x5d) [0x7f2aa09d029d]
1:  8  /apps/GPP/LIBFABRIC/1.21.0/GCC/lib/libfabric.so.1(+0xc8039) [0x7f2aa0b84039]
1:  9  /apps/GPP/LIBFABRIC/1.21.0/GCC/lib/libfabric.so.1(+0xcc3ab) [0x7f2aa0b883ab]
1: 10  /gpfs/scratch/ehpc01/de340/maestro/sep09/MPI_issue_reproducer/./mpi_ofi() [0x404426]
1: 11  /gpfs/scratch/ehpc01/de340/maestro/sep09/MPI_issue_reproducer/./mpi_ofi() [0x405b23]
1: 12  /gpfs/scratch/ehpc01/de340/maestro/sep09/MPI_issue_reproducer/./mpi_ofi() [0x4035e6]
1: 13  /gpfs/scratch/ehpc01/de340/maestro/sep09/MPI_issue_reproducer/./mpi_ofi() [0x4036cd]
1: 14  /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/compiler/latest/linux/compiler/lib/intel64_lin/libiomp5.so(__kmp_invoke_microtask+0x93) [0x7f2a9e54e493]
1: 15  /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/compiler/latest/linux/compiler/lib/intel64_lin/libiomp5.so(__kmp_fork_call+0x424) [0x7f2a9e4bcca4]
1: 16  /gpfs/apps/MN5/GPP/ONEAPI/2023.2.0/compiler/latest/linux/compiler/lib/intel64_lin/libiomp5.so(__kmpc_fork_call+0x1a3) [0x7f2a9e474d23]
1: 17  /gpfs/scratch/ehpc01/de340/maestro/sep09/MPI_issue_reproducer/./mpi_ofi() [0x40321b]
1: 18  /lib64/libc.so.6(+0x3feb0) [0x7f2a9e221eb0]
1: 19  /lib64/libc.so.6(__libc_start_main+0x80) [0x7f2a9e221f60]
1: 20  /gpfs/scratch/ehpc01/de340/maestro/sep09/MPI_issue_reproducer/./mpi_ofi() [0x402405]
1: =================================
srun: error: gs13r3b53: task 1: Aborted