part-persist: implement message aggregation

[AxelSchneewind]

Created a component for partitioned communication that supports message aggregation, based on the existing part-persist component. The goal is to prevent drops in effective bandwidth when using too fine-grained partitionings.

This component allows enforcing hard limits on size and count of transferred partitions, regardless of the partitioning required by the application. These limits can be specified using mca-parameters (min_message_size, max_message_count). Their default values might require revision.

If a user-provided partitioning violates the constraints, a more coarse-grained partitioning is selected, where multiple user-partitions are mapped to an internal (transfer) partition. Transfer of an internal partition is started as soon as Pready has been called on all corresponding user-partitions. Each transfer partition is associated with an atomic counter, tracking the number of corresponding user-partitions that have been marked ready.

The implementation could also be extended to use more advanced aggregation algorithms.

This implementation is a result of "Benchmarking the State of MPI Partitioned Communication in Open MPI" presented at EuroMPI 2024 (https://events.vsc.ac.at/event/123/page/341-program).

[janjust]

@mdosanjh Any chance we could get an initial review from you on this PR?

[mdosanjh]

I’ll look into it today.

…

On Tue, Jan 28, 2025 at 09:15 Tommy Janjusic ***@***.***> wrote: @mdosanjh <https://github.com/mdosanjh> Any chance we could get an initial review from you on this PR? — Reply to this email directly, view it on GitHub <#13039 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD4RGBD4MSPZA3QM3Y4HJZ32M6UJNAVCNFSM6AAAAABVHNG5I6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJZGQ2TIMZRGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hppritcha]

i think someone needs to look at the mpi4py failure. looks specific to changes in this PR.

[cniethammer]

Just an assumption from my side having a look at the error message without diving into the mpi4py code: It could be related to the extension of fields in the Request handle, which is mapped to a python object in mpi4py.

[hppritcha]

I doubt that. Probably failing only with mpi4py because it has the most extensive test suite in our CI infrastructure. This pr should not be merged till mpi4py passes.

[hppritcha]

If its any help here's where the segfault is occurring:

514	mca_part_persist_aggregated_start(size_t count, ompi_request_t** requests)
515	{
516	    int err = OMPI_SUCCESS;
517	    size_t _count = count;
518	
519	    for(size_t i = 0; i < _count && OMPI_SUCCESS == err; i++) {
520	        mca_part_persist_aggregated_request_t *req = (mca_part_persist_aggregated_request_t *)(requests[i]);
521	
522	        // reset aggregation state here
523	        if (MCA_PART_PERSIST_AGGREGATED_REQUEST_PSEND == req->req_type) {
524	            mca_part_persist_aggregated_psend_request_t *sendreq = (mca_part_persist_aggregated_psend_request_t *)(req);
525	            part_persist_aggregate_regular_reset(&sendreq->aggregation_state);
526	        }
527	
528	        /* First use is a special case, to support lazy initialization */
529	        if(false == req->first_send)
530	        {
531	            if(MCA_PART_PERSIST_AGGREGATED_REQUEST_PSEND == req->req_type) {
532	                req->done_count = 0;
533	                memset((void*)req->flags,0,sizeof(int32_t)*req->real_parts);
534	            } else {
535	                req->done_count = 0;
536	                err = req->persist_reqs[0]->req_start(req->real_parts, req->persist_reqs);
537	                memset((void*)req->flags,0,sizeof(int32_t)*req->real_parts);
538	            }
539	        } else {
540	            if(MCA_PART_PERSIST_AGGREGATED_REQUEST_PSEND == req->req_type) {
541	                req->done_count = 0;
542	                for(size_t j = 0; j < req->real_parts && OMPI_SUCCESS == err; j++) {
543	                    req->flags[j] = -1;
544	                }
545	            } else {
546	                req->done_count = 0;
547	            }
548	        }
549	        req->req_ompi.req_state = OMPI_REQUEST_ACTIVE;
550	        req->req_ompi.req_status.MPI_TAG = MPI_ANY_TAG;

at line 533. The problem is req->flags is NULL.

[AxelSchneewind]

Thank you very much, I will look into it. That error never occurred with my tests

[github-actions]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

da974db: use MCA_BASE_COMPONENT_INIT

• check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

[github-actions]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

7e95464: add aggregation scheme header to local_sources

• check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

[AxelSchneewind]

The bug that was found by the mpi4py-tests pipeline should now be resolved.
The error was caused by a mismatch between function pointers for the implementations of MPI_Start and MPI_P[send|recv]_Init (for more detail, see below).

I solved it by using separate request lists for each component.

This might also be relevant for PR #12796 that also introduces a new part-component.
I understand the issue as follows:

• The function pointer for MPI_Psend_Init is stored in the mca_part-object and set by mca_part_base_select.
• The function pointer for MPI_Start is stored in the request objects (req_start-field), which are preallocated when opening a component. As I understand it, all components (i.e. both part_persist and part_persist_aggregated) are opened during MPI_Init, and initialize some request objects each.

The segfault can occur when a request object is used where req_start points to the implementation from a component that has not been selected

[AxelSchneewind]

Could anyone take a look at this PR?

Also, the pipeline should be rerun as the timeout seems unrelated to the changes in this PR.