osc/ucx does not honor thread level requested

[edgargabriel]

The osc/ucx component does not honor the thread level requested when calling MPI_Init_thread. The main culprit is the line here:

https://github.com/open-mpi/ompi/blob/main/opal/mca/common/ucx/common_ucx_wpool.c#L58

which initializes the worker used by the osc component to be always UCS_THREAD_MODE_SINGLE.
This issue didn't surface previously because prior to ucx 1.18.0 the check for which thread is executing an operation was incomplete. However, starting from this commit in UCX

openucx/ucx@bdd207e

we are now getting errors of

ucp_worker.c:3055 Assertion `ucs_async_check_owner_thread(&(worker)->async)

when executing MPI_Put/Get calls in a thread other than the main thread, despite of having initialized MPI with THREAD_MULTIPLE support. A simple workaround is to configure UCX with --disable-assertions, but this is still an error in my opinion.

[edgargabriel]

Given that the offending line is in a block

if (opal_common_ucx_thread_enabled || ...) {
}

I am wondering whether the solution would be as simple as changing the assignment in L58 from UCS_THREAD_MODE_SINGLE to UCS_THREAD_MODE_MULTI, I can confirm that this fixes the issue.

Alternatively, I have experimented with code sequence where we extract the thread level from the ucp_context, e.g.

  ucp_context_query(wpool->ucp_ctx, &attr);
  worker_params.thread_mode = attr.thread_mode; 

but that only worked if we set in addition osc_ucx_enable_wpool_thread_multiple = true, which I think should not be required if you initialize MPI requesting MPI_THREAD_MULTIPLE support