Reduced Precision Abstraction (TF32, FP16/BF16 ..)

[mehmetyusufoglu]

Decreased precision data types like FP16 increases the performance for compute bound tasks like Matrix Multiplication, Dense Linear Algebra, Convolutions.

May be adding a general abstraction could be usefull for alpaka?

Especially transform and transformReduce functions in alpaka3 seem tobe good candidates for reduced precision types because they're templated and can work with custom data types.

[psychocoderHPC]

Yes we should support these data types. In principle, they should work out of the box but this should be tested.
I want to avoid an abstraction above these types because IMO this will be hard to maintain. If, after testing these types naturally, we still could re-evaluate whether an abstracted type is required.

[mehmetyusufoglu]

Some findings after a trial and research:

• Reduced types turned out to be effective only if Tensor Cores are used.
• For using Tensor Cores EITHER cuBLAS or cuCNN specific functions should be used (e.g cublasGemmEx not cublasGemm) OR wmma instructions are needed. See TensorCores.
• For every reduced type (e.g. 16 bit type like FP16) implemented; the functionalities of Sizeof/align, mdspan strides, memcpy operations should be tested.

[mehmetyusufoglu]

C++23 new floating point types:
https://en.cppreference.com/w/cpp/types/floating-point.html#Fixed_width_floating-point_types