Very slow __pthread_mutex_unlock on writemsg on EFA

[abcdabcd987]

I'm trying to improve 8 KiB message WRITE rate on single EFA device on p5en. Currently I'm getting 2.2 Mpps (around 140 Gbps) on single thread. I've pinned the CPU on the correct NUMA node, and allocated data structures on the pinned CPU (so hopefully they are allocated on the memory of the same NUMA node).

I ran a perf on my program and generated the following flame graph:

To my surprise, __pthread_mutex_unlock takes very long time underneath efa_rdm_rma_writemsg. I don't expect this because it's single threaded. I'm curious about why this would happen.

Alternatively, I'm also trying out efa-direct fabric to skip the lock.

Environment:

• libfabric 2.1.0
• CFLAGS for compiling libfabric: -g -O2 -fno-omit-frame-pointer
• p5en
• Using single EFA device
• Using single thread

[shijin-aws]

Thanks for reaching out, though it worth looking at the slowness of mutex unlock, for single thread, can you specify domain_attr.threading = FI_THREAD_DOMAIN in your hints? I believe the mutex lock you refer to is https://github.com/ofiwg/libfabric/blob/main/prov/efa/src/rdm/efa_rdm_rma.c#L493. It will become a no-op for the domain thread level.

[shijin-aws]

efa-direct acquire the ep lock https://github.com/ofiwg/libfabric/blob/main/prov/efa/src/efa_rma.c#L217 which is a mutex lock for FI_THREAD_SAFE as well. In order to remove such locking for single threaded application you can also use FI_THREAD_DOMAIN

[abcdabcd987]

Thanks for the fast response! Setting FI_THREAD_DOMAIN bumps up the speed from 2.199 Mpps to 2.316 Mpps. Really appreciate it!

BTW, do you know what's the hardware spec for small packet rate? At my current rate, it's roughly 152 Gbps (76% of 200 Gbps) in terms of bandwidth. I guess the hardware could get full bandwidth at MTU (8928 bytes)?

[shijin-aws]

You can run ib_write_bw perftest to verify https://github.com/linux-rdma/perftest , my collection was that it won't get full bandwidth at 8K. I may be wrong.

[shijin-aws]

Like

# RDMA responder
ib_write_bw -c SRD -x 0 -s 8192 -n 128 -Q 1 -F -p 50000 -l 64 --report_gbits --inline_size 0
# RDMA initiator
ib_write_bw -c SRD -x 0 -s 8192 -n 128 -Q 1 -F -p 50000 -l 64 --report_gbits --inline_size 0 <responder ip>

[abcdabcd987]

Those commands are super helpful! Thank you!

I made a small change (increase iters, specify EFA device and GPU device):

./ib_write_bw -c SRD -x 0 -s 8192 -Q 1 -F -p 50000 -l 64 --report_gbits --inline_size 0 -d rdmap85s0  -n 131072 --use_cuda=0
./ib_write_bw -c SRD -x 0 -s 8192 -Q 1 -F -p 50000 -l 64 --report_gbits --inline_size 0 -d rdmap162s0 -n 131072 --use_cuda=7 127.0.0.1

Here's the result:

 WARNING: BW peak won't be measured in this run.
initializing CUDA
Listing all CUDA devices in system:
CUDA device 0: PCIe address is 59:00
CUDA device 1: PCIe address is 5A:00
CUDA device 2: PCIe address is 72:00
CUDA device 3: PCIe address is 73:00
CUDA device 4: PCIe address is 8B:00
CUDA device 5: PCIe address is 8C:00
CUDA device 6: PCIe address is A4:00
CUDA device 7: PCIe address is A5:00

Picking device No. 7
[pid = 3096854, dev = 7] device name = [NVIDIA H200]
creating CUDA Ctx
making it the current CUDA Ctx
CUDA device integrated: 0
allocated GPU buffer of a 16384 address at 0x55a8e13fdf50 for type CUDA_MEM_DEVICE
---------------------------------------------------------------------------------------
                    RDMA_Write Post List BW forwarding Test
 Dual-port       : OFF          Device         : rdmap162s0
 Number of qps   : 1            Transport type : Unknown
 Connection type : SRD          Using SRQ      : OFF
 PCIe relax order: ON           Lock-free      : OFF
 ibv_wr* API     : ON           Using DDP      : OFF
 TX depth        : 128
 Post List       : 64
 CQ Moderation   : 1
 CQE Poll Batch  : 16
 Mtu             : 4096[B]
 Link type       : IB
 Buffer size     : 8192[B]
 GID index       : 0
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 000000 PSN 0x1ce1bf RKey 0x700077 VAddr 0x007f6c09e02000
 GID: 254:128:00:00:00:00:00:00:16:42:59:255:254:129:69:103
 remote address: LID 0000 QPN 000000 PSN 0x4c76c8 RKey 0x9000fa VAddr 0x007fda89e02000
 GID: 254:128:00:00:00:00:00:00:16:243:246:255:254:54:232:197
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 8192       131072           0.00               139.47               2.128208
---------------------------------------------------------------------------------------
deallocating GPU buffer 00007f6c09e00000
destroying current CUDA Ctx

Looks like perftest is slightly slower than my program. I guess maybe this is very close to hardware limit? I'll probably experiment with efa-direct as well.