Topology caching in hwloc_topology_load

[fsimonis]

Hi,
I propose to integrate an easy built-in way to cache the topology file to speed up the load of hwloc_topology_load using a single env such as HWLOC_TOPOLOGY_CACHE=1. This could maybe even be enabled by default.

Currently, this can be done manually by running:

lstopo --of xml topology.xml
export HWLOC_XMLFILE=$( readlink -f topology.xml )

However, it would be nice to have a simple switch without needing to run a separate tool. This way hwloc could choose a most appropriate location for the cache such as a tmpfs that clears between boots.

Motivation:
I'm dealing with a test suite of over 1k MPI tests. The topology detection inside OpenMPI MPI_Init > PMIx_Init > pmix_hwloc_setup_topology takes the majority of time.
Using the export above, I can reduce the time spend in MPI_Init from 108ms to 6ms.
The detection eats up a proud 102s for each of our 10 test configurations, which is a bit extreme.

[bgoglin]

Hello.

What's complicated about enabling this across different applications is that the topology depends on some configuration flags given at runtime, and also on some environment variables. Most of these are about enabling/disabling things. In those cases, you just enable everything in the cache and then filter things out later (including cgroups, etc). However there are corner cases where it really changes some things, hence an automatic solution would have to make sure the cache gets disabled whenever something is set to non-default and contradicts the cache.

So I think I would rather keep it disabled by default. Also having hwloc decide of the XML cache filename would require to basically encode the configuration in the filename to avoid loading the wrong cache too. I'd rather have the user choose its own filename instead.

So basically HWLOC_TOPOLOGY_CACHE=/path/to/my/file.xml exports that file if it doesn't exist yet, and loads it otherwise (and we'll have to be careful that multiple processes may try to create it at the same time in a MPI run).

By the way, another way to speed up things is to use shmem topologies. This was designed for saving memory on manycores by loading a single topology per node. But it may actually be used by consecutive runs too. Howeber it would require some work on the MPI side.