Convert from HIP to hipper

[joaander]

Description

Replace all HIP calls with hipper calls. Continues #427.

Motivation and context

HIP is complex, out of our control, gets in the way, and often breaks things. There is no need to use it for CUDA builds. hipper is a thinner translation layer that @mphoward developed that works around these issues by using CUDA more directly than HIP for CUDA builds and falls back to HIP for AMD builds.

[mphoward]

When we do this, we should make sure to look at the CMake setup too. Right now, there is a lot of complicated code needed to trick CMake into using HIP as CUDA. It would be nice if we could only take this path when HIP is being used to do the compilation.

[joaander]

When we do this, we should make sure to look at the CMake setup too. Right now, there is a lot of complicated code needed to trick CMake into using HIP as CUDA. It would be nice if we could only take this path when HIP is being used to do the compilation.

Yes. We may also need to refactor the HIP/CUDA CMake code considerably in for this issue and #1101. This code was written for very early ROCm/HIP tools and has not been maintained over the years.

I don't have access to an AMD GPU system for testing yet. OLCF's test system is not yet open to INCITE projects and NCSA Delta (which will have only one AMD GPU node) has been delayed again. If these are still not available when I start working on this, I will make the conversion testing only on CUDA and then make the changes needed for AMD support later.

[mphoward]

I will make the conversion testing only on CUDA and then make the changes needed for AMD support later.

I support this. I also do not have an AMD system to test on, and ROCm/HIP has been unstable, at least in the past.

[joaander]

Revisiting this: It will be a significant effort to port HOOMD to hipper and may require updates to hipper itself. I need to look more into this before proceeding.

However, the alternative is to continue using only HIP. Current versions of HIP are no longer header only and require a build and install step. I find AMD's compilation documentation severely lacking. I can not expect the majority of HOOMD users to follow it. Additionally, HIP has not been updated on conda-forge for several years. This alternative will therefore require that we learn how to build and install HIP, document it for our users, and maintain a conda-forge package.

[mphoward]

I admittedly have not been trying to keep hipper up to date because (1) the features we are actually using are pretty minimal and (2) I don’t have any AMD GPUs for testing. We could be more active in this if we want to pursue using it throughout HOOMD.

If a conversion is going to be made, have you given any thought to Intel oneAPI? I haven’t tried it so I’m not sure what use/performance is actually like. That is probably even more work, though, I would imagine.

Using only HIP is a little more palatable now that the CMake build system is fixed, but I agree that the documentation is general very poor.

[joaander]

I also have not tried oneAPI, but I have spoken with people who have. It is a vastly different programming model to CUDA/HIP. In addition to rewrites of all kernels, it would require a complete overhaul of the memory management system as oneAPI requires the use of its provided memory management classes. I got conflicting answers on whether there was any possibility of oneAPI / CUDA interoperability. One knowledgeable individual indicated that it was not possible at all, requiring us to do a complete port or none at all.

oneAPI also has no support for zero-copy interoperability with Python at this time - which is one of the most popular features of v3.

For Intel GPU support, there is a third party package that implements HIP on Intel. The large DOE centers have an interest to support projects like that.

If oneAPI gains traction in the long run and replaces HIP in the community, we will need to consider a port then. At present, a oneAPI port would be require a massive time investment and would remove functionality.

Switching to hipper in the meantime will be time consuming, but not unduly so. If I convert to hipper, I can remove the outdated HIP and hipCUB submodules. It is only a matter of time before one of those runs into compatibility issues with a new CUDA version.

[mphoward]

OK! That all makes sense. oneAPI would be an enormous amount of work then, and it’s unclear how much traction it will have.

I favor the hipper approach, then, since it will allow true CUDA builds without any dependencies we have to maintain. It also means we don’t have to teach how to compile HIP.

[jglaser]

Please let me know if you need me to test any code on AMD.