prov/efa: Track mr status #11146
Draft
prov/efa: Track mr status #11146
+144
−22
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, there is no track for the reg/dereg status of an MR in efa provider, because `fi_close(mr)` will simply free all the `efa_mr` object. This will cause crash or other undeterministic when an application close a mr when there are outstanding operations still reference it. This patch resolves this problem by introducing two fields in efa_mr structure: active and ref. ref is the count of operations (op entries) that an mr is referenced. It is initialized as 0 during mr registration, , incremented when the mr is referenced by an operation, and decremented when the operation is resolved. active is a simple boolean that is toggled as true when mr is registered and toggled as false when mr is deregistered by fi_close. The lifecycle or an efa_mr object behaves as: 1. Allocated during fi_mr_reg*, initialize active as 1, ref as 0 2. op 1 referenced the mr, ref++ 3. op 2 referenced the mr, ref-- 4. fi_close dereg the mr, active is set as 0, while the object is not freed. 5. op 1 completes, ref-- 6. op 2 completes, ref--, now ref is 0 and active is 0, object is freed. The sequence of 4, 5, 6 can be any order, and the object will be freed only when active is 0 && ref is 0 Also introduce a check in the operation calls that any access to inactive mr will fail with -FI_EINVAL. Signed-off-by: Shi Jin <[email protected]>
When the pkt send/write call failed, the tx pkt should be released, because it was removed from the queued_pkts dlist already. Otherwise it will not be cleaned in the iteration of queued_pkts in the txe/rxe error handling and causes assertion errors in tx pkt pool destroy Signed-off-by: Shi Jin <[email protected]>
For tx pkt, pkt_entry->entry is linked to either peer->outstanding_tx_pkts or txe->queued_pkts. It must be removed from the list as part of the pkt entry release, otherwise it will cause double free bug when txe_handle_err and txe_release are both called for the same txe. Signed-off-by: Shi Jin <[email protected]>
Signed-off-by: Shi Jin <[email protected]>
The rx pkt in this error path may already be freed. Temporarily skip the release to avoid double free. Signed-off-by: Shi Jin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, there is no track for the reg/dereg status of
an MR in efa provider, because
fi_close(mr)will simplyfree all the
efa_mrobject. This will cause crash or otherundeterministic when an application close a mr when there
are outstanding operations still reference it.
This patch resolves this problem by introducing two fields
in efa_mr structure: active and ref.
ref is the count of operations (op entries) that an mr
is referenced. It is initialized as 0 during mr registration,
, incremented when the mr is referenced by an operation,
and decremented when the operation is resolved.
active is a simple boolean that is toggled as true when
mr is registered and toggled as false when mr is deregistered
by fi_close.
The lifecycle or an efa_mr object behaves as:
not freed.
freed.
The sequence of 4, 5, 6 can be any order, and the object will be
freed only when active is 0 && ref is 0
Also introduce a check in the operation calls that any access to
inactive mr will fail with -FI_EINVAL.