libfabric "patch" routines corrupt the procedure linkage table (PLT) on AARCH64 under Cray MPICH MPI_Init

[paboyle]

Describe the bug

During MPI_Init call on ALPS at CSCS Switzerland, AARCH64 NVIDIA Grace/H200 libfabric "patch" routines corrupt
the procedure linkage table (PLT).

To Reproduce

Call MPI_Init but symptom can be non-obvious and deferred.

Expected behavior
Program runs to completion.

Output
MPI_Comm_dup fails, after MPI_Init, with asm level single step showing the step through the
dylib PLT has the "wrong" instruction br x15 at the entry point for MPI_Comm_dup in PLT.

Putting a watch point on the address of the PLT entry and running call to MPI_Init traps the PLT being
corrupted by a call stack I will embed below.

Environment:
Linux AARCH64 NVIDIA Grace/H200 superchip node with HPE Slingshot.

Additional context

[paboyle]

Call stack at time of corruption:

watch *0x408ed0

[paboyle]

Post corruption PLT entry:

layout asm
info registers

[paboyle]

Prior to MPI init, PLT entry is verbatim as in disassembly:

(gdb) p/x *((uint32_t *)0x408ed0)
$2 = 0x90001750

After MPI_Init the address is corrupt:

(gdb) p/x *((uint32_t *)0x408ed0)
$3 = 0xd61f01e0

Disassembly of binary: intended instruction at x408ed0 is the value prior to calling MPI_Init.
The OFI patch process has caused massive corruption of dylib hooks in this PLT section.

[paboyle]

End result was a segmentation fault when the random or stale value (0x10) in register x15 was branched to by the post-patch call to MPI_Comm_dup.

[paboyle]

Software versions:

cray-mpich-8.1.32
/opt/cray/libfabric/1.22.0/lib64/libfabric.so.1

[paboyle]

The opcode that was written (0xd61f01e0) into the address looks to have come from
the routine "br" in

https://github.com/ofiwg/libfabric/blob/main/prov/util/src/util_mem_hooks.c

static uint32_t br(unsigned int reg)
{
	return (0xd61f << 16) + (reg << 5);
}

[paboyle]

For completeness, problem is most easily studied on single rank or single node
by setting

MPICH_SINGLE_HOST_ENABLED=0

This triggers/forces the CXI / fabric initialization under Cray-MPICH even on single node.

Otherwise the problem only arises under multiple node jobs (and as this is through the batch system, it's hard to get a debugger running on them with interactive control).

[paboyle]

Setting either

export FI_MR_CACHE_MONITOR=kdreg2

OR

export FI_MR_CACHE_MAX_COUNT=0

removes the problem, presumably disabling the memhook code path.

[swelch]

Thanks, Peter. I have opened an internal HPE JIRA to track this issue.

[sielicki]

can you show the link order? eg: ldd output.

sielicki@5fddedd may be a fix.