⚠ Help review okina for mfem-4.0

[tzanio]

MFEM Developers,

The okina branch (PR #631) introduces several new major features in the library, including GPU support, device memory management and partial assembly in BilinearForm. These new features will form the backbone of the upcoming mfem-4.0 release.

Please spend some time with the branch this week to familiarize yourself with the changes and to provide feedback on the pull request.

To help you in the review, below is a summary of the main new features.

Examples

• Currently Example 1, Example 6 and Laghos have been ported to okina.
• See ex1.cpp, ex1p.cpp, ex6.cpp and ex6p.cpp to get an idea of the changes in the user API.

GPU support

• Hardware devices, such as GPUs, and programming models, such as CUDA, OCCA, RAJA and OpenMP are abstracted in the Device class, see general/device.?pp.
• Device kernel definition and execution are provided by the okina interface in general/okina.hpp, general/cuda.?pp and general/occa.?pp.
• Okina = Okina Kernel Interface for Numerical Analysis
• The class mm in general/mm.?pp provides lightweight host/device memory manager designed to work seamlessly with the new device kernels.
• Simple vector and tensor classes for dense linear algebra on the device are implemented in linalg/dtensor.hpp.

GPU-enabled kernels

The following files contain okina kernels based on the new MFEM_FORALL macro:

• in the linalg/ directory: vector.cpp, operator.cpp, densemat.cpp and sparsemat.cpp.
• in the fem/ directory: bilinearform_ext.cpp and bilininteg_ext.cpp.

Partial assembly classes

The following files (all under fem/) add support for partial assembly in the core library.

• bilinearform.?pp, bilinearform_ext.?pp and pbilinearform.?pp include:
  • new constructors with an AssemblyLevel and element_batch argument
  • the FABilinearFormExtension (full assembly), EABilinearFormExtension (element assembly), PABilinearFormExtension (partial assembly) and MFBilinearFormExtension (matrix-free) classes
  • new versions of FormLinearSystem and FormSystemOperator that return an Operator
• bilininteg.?pp, bilininteg_ext.?pp and diffusion.okl implement mass and diffusion PA integrators.

How can you help?

• Review and provide feedback for any of the above groups of changes
• Run tests with CUDA, RAJA, OCCA, etc. on Linux, Mac, LC (e.g. Lassen) and report issues
• Document the new features, e.g. in the Partial assembly classes
• Add some FA, EA or MF kernels (currently only PA kernels are implemented)
• Add support for element batching (currently ignored)
• Add NVCC support in the CMake build system
• ...

[tzanio]

See also the mfem-4.0 roadmap in: #813