libfabric interactions with rdma-core provider

[alokprasad]

I have doubt regarding provider for EFA, i was under impression that libfabric uses prov/efa for low level access..
but this commit in

    commit a2b6712190d59135cb711fcf5c06a5806efdbc36
    Author: Gal Pressman <[email protected]>
    Date:   Tue Apr 16 12:10:11 2019 +0300
    prov/efa: Use efa libibverbs provider
    Use the existing libibverbs EFA provider for device/kernel access.

    The configure process now requires passing the path to rdma-core build
    directory in order to enable the EFA provider.
    For example:
    $ ./configure --enable-efa="/home/gal/rdma-core/build"

    Signed-off-by: Gal Pressman <[email protected]>
    Signed-off-by: Robert Wespetal <[email protected]>
    ```
    description conveys the libfabric uses provider under
    rdma-core/tree/master/providers/efa
    
    Question:
    1.Is rdma-core required for RDMA device or EFA for user app.?
    2.what is difference between providers in rdma-core and in libfabric
    
    

[shijin-aws]

1.Is rdma-core required for RDMA device or EFA for user app.?

Libfabric EFA provider depends on rdma-core since libfabric 1.10. You can install rdma-core from source or use the packages from AWS EFA installer

2.what is difference between providers in rdma-core and in libfabric

They are finally the same EFA device, but different userspace API interfaces. Libfabric EFA provider support more capabilities and semantics that verbs EFA provider won't support, such as tagged message, msg ordering, etc

[alokprasad]

@shijin-aws thanks for clarifying , so can i say we two flows
1.Libfabric API - > prov/efa( libfabric) -> EFA Device
2.Libfabric API-> providers/efa( rdma-core)-> EFA Device

or its something like Libfabric API -> providers/efa ( rdma-core) -> prov/efa (libfabric) -> EFA Device
rdma-core provider - does low h/w intiailization
libfabric provider -> all other things.

based on what type of usage and API used.

[j-xiong]

Although both are called "provider", they are different entities at different layers of the software stack.

Libfabric providers are libfabric API implementations for a specific HW/fabric. RDMA-core providers are Verbs API implementations for a specific HW/HCA and is sometimes called "UMD" (User Mode Driver).

For EFA, the flow looks more like this:

Libfabric API --> libfabric efa provider --> Verbs API --> rdma-core efa provider --> EFA device.

[alokprasad]

@j-xiong Thanks for clarification , so for supporting new hardware with libfabric its like mandatory for developing both providers

• one libfabric provider and other rdam-core provider .

[j-xiong]

Yes if the new hardware needs rdma-core (i.e., needs to support Verbs). Many libfabric providers don't need rdma-core at all, e.g., cxi, tcp, psm2.

[shijin-aws]

Actually libfabric EFA provider starts removing rdma-core dependency on data path 741ca49

[alokprasad]

@j-xiong Thanks , my another question to get better clarity (maybe UEC will right forum ) will same thing design will be done for UEC -
rdma-core for verbs support and then libfabric leveraging those verbs api. or there will no need to rdma-core providers like you mentioned above.

[shijin-aws]

UEC only has libfabric as the software gateway above the transport layer, libfabric provider is not required to leverage verbs API

[j-xiong]

Correctly, UE vendors have the freedom to implement their libfabric providers as long as the specifications are followed. Any question beyond the published the specifications (v1.0) should be directed to the UEC workgroups.