prov/shm getting performance issue in system to cuda memory transfer

[mlefebvre1]

Hello there,

We're evaluating libfabric as an abstraction layer for inter-host and intra-host video frame memory transfers. During performance comparisons between libfabric and native library calls (e.g., cudaMemcpy), we've observed unexpected performance characteristics specifically when transferring intra-host from system memory to CUDA memory. The reverse direction (CUDA to system memory) performs as expected.

Configuration Details

OS: Linux 6.8.0-79-generic #79-Ubuntu SMP PREEMPT_DYNAMIC T x86_64 GNU/Linux
Version: libfabric v2.2.0
Provider: shm
Transfer method: Remote write with immediate data via fi_writemsg
Protocol: We're not using CMA or XPMEM, so I assume it's using SAR, at least that's what the call stack shows.
Completion handling: Tested both busy-wait polling (fi_cq_read) and fi_cq_sread - no latency difference observed between approaches
Process 1 registers system memory, process 2 registers CUDA memory. For system to CUDA transfers, process 1 is the initiator and process 2 is the target. For CUDA to system transfers, process 1 is the target and process 2 is the initiator

Problem

CUDA-to-system memory transfers perform as expected and show comparable results to native calls. However, system-to-CUDA memory transfers exhibit unexpected performance degradation, suggesting an issue with this specific transfer direction when using the shm provider. Here's the results:

System-to-CUDA

TransferSize(Bytes)	Latency(us)	CPU Usage	Library
2547840	627	0.04	libfabric
2547840	173	0.01	cuda
22389120	5341	0.32	libfabric
22389120	850	0.05	cuda

CUDA-to-system

TransferSize(Bytes)	Latency(us)	CPU Usage	Library
2547840	184	0.014	libfabric
2547840	182	0.014	cuda
22389120	850	0.055	libfabric
22389120	834	0.056	cuda

We've also attempted to reproduce the result with fabtests fi_rma_bw and fi_rma_pingpong, but we get mixed results, which you might have an explanation for.

fi_rma_bw:

$ fi_rma_bw -p shm -s test1 -S 2547840 -c sread  -o writedata -D cuda`
bytes   iters   total       time     MB/sec    usec/xfer   Mxfers/sec
2.4m    200     485m        0.13s   4017.53     634.18       0.00

$ fi_rma_bw -p shm -S 2547840 -c sread -o writedata  test1
bytes   iters   total       time     MB/sec    usec/xfer   Mxfers/sec
2.4m    200     485m        0.12s   4216.11     604.31       0.00

$ fi_rma_bw -p shm -s test1 -S 2547840 -c sread  -o writedata
bytes   iters   total       time     MB/sec    usec/xfer   Mxfers/sec
2.4m    200     485m        0.03s  18179.38     140.15       0.01

$ fi_rma_bw -p shm -S 2547840 -c sread -o writedata  -D cuda test1
bytes   iters   total       time     MB/sec    usec/xfer   Mxfers/sec
2.4m    200     485m        0.03s  18180.68     140.14       0.01

fi_rma_pingpong:

$ fi_rma_pingpong -p shm -s test1 -S 2547840 -c sread -o writedata -D cuda
bytes   iters   total       time     MB/sec    usec/xfer   Mxfers/sec
2.4m    100     485m        0.07s   7093.49     359.18       0.00

$ fi_rma_pingpong -p shm -S 2547840 -c sread -o writedata test1
bytes   iters   total       time     MB/sec    usec/xfer   Mxfers/sec
2.4m    100     485m        0.07s   7093.49     359.18       0.00

$ fi_rma_pingpong -p shm -s test1 -S 2547840 -c sread -o writedata
bytes   iters   total       time     MB/sec    usec/xfer   Mxfers/sec
2.4m    100     485m        0.07s   7099.52     358.88       0.00

$ fi_rma_pingpong -p shm -S 2547840 -c sread -o writedata -D cuda test1
bytes   iters   total       time     MB/sec    usec/xfer   Mxfers/sec
2.4m    100     485m        0.07s   7099.62     358.87       0.00

fi_info dump:

    caps: [ FI_RMA, FI_REMOTE_WRITE, FI_MULTI_RECV, FI_LOCAL_COMM, FI_RMA_EVENT, FI_HMEM ]
    mode: [  ]
    addr_format: FI_ADDR_STR
    src_addrlen: 20
    dest_addrlen: 0
    src_addr: fi_ns://test1:10001
    dest_addr: (null)
    handle: (nil)
    fi_tx_attr:
        caps: [ FI_RMA, FI_READ, FI_WRITE, FI_HMEM ]
        mode: [  ]
        op_flags: [  ]
        msg_order: [ FI_ORDER_SAS ]
        inject_size: 0
        size: 1024
        iov_limit: 4
        rma_iov_limit: 4
        tclass: 0x0
    fi_rx_attr:
        caps: [ FI_RMA, FI_REMOTE_WRITE, FI_MULTI_RECV, FI_RMA_EVENT, FI_SOURCE, FI_HMEM ]
        mode: [  ]
        op_flags: [  ]
        msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, FI_ORDER_ATOMIC_WAW ]
        size: 1024
        iov_limit: 4
    fi_ep_attr:
        type: FI_EP_RDM
        protocol: FI_PROTO_SHM
        protocol_version: 1
        max_msg_size: 18446744073709551615
        msg_prefix_size: 0
        max_order_raw_size: 0
        max_order_war_size: 0
        max_order_waw_size: 0
        mem_tag_format: 0xaaaaaaaaaaaaaaaa
        tx_ctx_cnt: 1
        rx_ctx_cnt: 1
        auth_key_size: 0
    fi_domain_attr:
        domain: 0x0
        name: shm
        threading: FI_THREAD_SAFE
        progress: FI_PROGRESS_MANUAL
        resource_mgmt: FI_RM_ENABLED
        av_type: FI_AV_UNSPEC
        mr_mode: [ FI_MR_VIRT_ADDR, FI_MR_HMEM ]
        mr_key_size: 8
        cq_data_size: 8
        cq_cnt: 1024
        ep_cnt: 256
        tx_ctx_cnt: 1024
        rx_ctx_cnt: 1024
        max_ep_tx_ctx: 1
        max_ep_rx_ctx: 1
        max_ep_stx_ctx: 0
        max_ep_srx_ctx: 0
        cntr_cnt: 0
        mr_iov_limit: 4
        caps: [ FI_LOCAL_COMM ]
        mode: [  ]
        auth_key_size: 0
        max_err_data: 0
        mr_cnt: 0
        tclass: 0x0
    fi_fabric_attr:
        name: shm
        prov_name: shm
        prov_version: 202.0
        api_version: 2.2
    nic: (nil)

Any ideas?

[aingerson]

Hi @mlefebvre1 ! Thanks for opening up an issue and for giving libfabric a try!
Yes, I think the issue here is IPC vs SAR - so when you go D->H, shm uses an IPC handle and then issues a single copy (read from the device) and that's it. But when you go H->D, we can't send a handle so we copy into a shm bounce buffer and then on the target issue a second copy to go to the target device memory. If this message size is large, we have to chunk the transfer into 32k segments and send each separately so for your case with 2MB, you're looking at doing this 78 times!
So essentially the paths are (D->H)x1 vs (H->H->D)x78.
The point is H->D for large messages is very slow... I've worked on a new shm architecture to solve this issue which allows for a receiver side fallback to IPC (so it would turn it into a single H->D write). We are targeting this new architecture for the next major release in December and that would include the IPC fallback protocol. Could you give this branch a try and let me know what you see with the new protocol to handle this case? https://github.com/aingerson/libfabric/tree/shm_new_ipc
Tagging @zachdworkin because he added the new fallback protocol

My best explanation for fabtests being inconsistent is two things:

• some tests do a one sided test and others do a two sided test (regular bw ie rdm bw is going to do a one sided bw from client to server while the rma bw test is going to do a two sided test so even if you specify D->H, you're actually going to be doing half D->H and half H->D)
• are you specifying different devices in your fabtests? I've known cuda do have issues with two processes accessing the same device simultaneously due to context switching on the cuda driver level. If you have more than one device, you can specify each one to use a different device by specifying the device index with the -i parameter. If you only have one device, you can use CUDA MPS https://docs.nvidia.com/deploy/mps/index.html which is basically a daemon that runs to limit the number of contexts you use and have both processes use the same context (from my understanding)

Let me know if you're able to test that branch and what you see!

[mlefebvre1]

Hello,

First of all, thank you for the previous response. It corroborates what I witnessed from flamegraphs and debug sessions.

I was able to perform the same tests with your shm_new_ipc branch. Unfortunately, I don't see any improvements in performance, it pretty much gives the same previous results. Maybe I'm missing some environment variables or configuration.

I can produce flamegraphs and inspect them if you can provide me with some pointers.

Thank you!

[aingerson]

Thanks for giving it a go! I'm really surprised that didn't do anything. Did you remove your old installation of libfabric entirely? The branch I gave you is a version behind so if you installed into the same location, your application is probably still going to pick up the newer version if it exists. Please verify you're running the new version.

I tried rebasing to upstream to pick up the latest version number and you can find that version in my shm_new_ipc2 branch. I did a quick run to see if it worked and it didn't crash so I think it was ok but there were some conflicts so note that it isn't 100% verified. Let me know if you have issues with that one too.

Thanks!

[mlefebvre1]

Okay, I'm not sure what I was doing wrong previously, but I re-installed libfabric with shm_new_ipc branch and I get much better results for H2D.

TransferSize(Bytes)	Latency(us)	CPU Usage	Library
2547840	192	0.019	libfabric
2547840	173	0.017	cuda
22389120	1230	0.08	libfabric
22389120	850	0.05	cuda

[aingerson]

@mlefebvre1 Thanks for double checking and rerunning! That difference still looks high.
How are you testing your CUDA latency (without libfabric)? Are you doing an IPC write with two processes or are you just doing a memcpy within the process?

[mlefebvre1]

I'm just doing a cudaMemcpy within a single process. I would not expect libfabric to copy the entire buffer host-to-host between the processes, instead just do a cudaMemcpy and pass the handle to the other process.

[aingerson]

That may have something to do with it but I'm not positive. From previous testing, we've noticed that there's a bigger difference between intraprocess H->D memcpy and IPC H->D write than there is between intraprocess D->H and IPC D->H read. To get a better side by side comparison, you'd have to modify your non-libfabric testing to try a cross-process IPC so for the D->H case it would be an IPC read from the target and for H->D it would be an IPC write from the source process. If you're willing to run that then we can root cause if it's the read/write issue. If that matches the raw cuda numbers then I think that's unfortunately the best we're going to get since copying cross-process H->D your two options are bouncing through the host for each chunk or do an IPC write. If that still doesn't match the raw implementation then I can look into it more and see what could be causing the extra overhead.

[mlefebvre1]

Hi,
I've resolved an issue in my test harness and re-ran the tests with the shm_new_ipc branch. The results now show no noticeable performance difference compared to native (cudaMemcpy).

Thanks again!

[aingerson]

@mlefebvre1 Oh amazing!! Thank you for retesting and letting me know! I am going to keep this open until the shm update is merged so we can track the issues that it will help fix.