POSIX latency increases rapidly after 64B even though message size is still < seg_size (bcopy path)

[TroyMitchell911]

Environment

MPI: Open MPI (pml ucx)
UCX: v1.19.0
Benchmark: osu-micro-benchmarks/mpi/pt2pt/osu_latency
Node count: 2
OS: Linux
Transport: posix

When running the OSU latency test with UCX_TLS=posix under MPI+UCX, I observe that latency stays flat before 64 bytes, but starts to grow sharply once the message size exceeds ~64B.
According to UCX’s design, this transition point corresponds to exceeding fifo_elem_size and switching to the bcopy path.
However, even when the message sizes remain well below seg_size, the latency still continues to increase as message size grows.

This behavior seems unexpected: if UCX is using bcopy and the message fits within a segment, I would expect latency to remain nearly flat.

I'm trying to understand whether this is expected behavior or a performance issue.

here is my command:

mpirun --allow-run-as-root -np 2 \
  -x UCX_TLS=posix \
  --mca pml ucx \
  --mca coll_ucc_enable 1 \
  --mca coll_ucc_priority 100 \
  --mca pml_ucx_tls any \
  --mca pml_ucx_devices any \
  /opt/mpitest/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

and there is the result:

# OSU MPI Latency Test v7.5
# Datatype: MPI_CHAR.
# Size       Avg Latency(us)
1                       0.94
2                       0.93
4                       0.92
8                       0.93
16                      0.92
32                      1.04
64                      1.05                      ---- prepare to change to bcopy
128                     1.73                     |
256                     1.79                     |
512                     2.80                     |       Why did the latency increase in this range, 
1024                    3.47                    |        while the short-am msg latency did not increase in its own range?
2048                    4.45                    |
4096                    6.26                    |
8192                    9.45                    ----- more than size of one bcopy
16384                  16.86
32768                  31.13
65536                  62.73
131072                126.36
262144                294.13
524288                542.43
1048576              1007.92
2097152              1927.02
4194304              3754.30

[TroyMitchell911]

Oh, when I try to modify FIFO_ELEM_SIZE, FIFO_SIZE and SEG_SIZE:

mpirun --allow-run-as-root -np 2 \
-x UCX_TLS=posix --mca pml ucx \
--mca coll_ucc_enable 1 --mca coll_ucc_priority 100 \
--mca pml_ucx_tls any --mca pml_ucx_devices any \
-x UCX_MM_FIFO_ELEM_SIZE=1536 -x UCX_MM_FIFO_SIZE=2048 -x UCX_MM_SEG_SIZE=16384 osu_latency

I got this result:

# OSU MPI Latency Test v7.5
# Datatype: MPI_CHAR.
# Size       Avg Latency(us)
1                       1.14
2                       0.97
4                       0.96
8                       0.96
16                      0.97
32                      1.16
64                      1.17
128                     1.39
256                     2.16
512                     2.21
1024                    2.37
2048                    6.00
4096                    7.52
8192                   10.72
16384                  23.95
32768                  37.30
65536                  61.49
131072                123.53
262144                280.79
524288                513.87
1048576               950.58
2097152              1811.58
4194304              3519.73

Therefore, latency also increases with short-am. So my question is, aren't both short-am and buffer-copy just fixed framework overhead plus memcpy_size? Why does memcpy_size have a microsecond-level impact?