Heterogeneous computing in spectral methods

[brownbaerchen]

I found an imperfect but very simple solution to the woes stemming from the small memory limit on GPUs.
After various improvements to the memory footprint, the remaining issue was that cusparse allocates loads of memory to do sparse matrix-matrix multiplication (spgemm) fast. And this exceeded the 40GB limit per device already for relatively small matrices.
Now, the only place where I need spgemm is before the LU factorizations. These are done on CPU anyways. So, now I keep copies of the matrices involved there on CPU and perform the spgemm before the LU factorizations on CPU. The GPU is faster in spgemm than the CPU, but a slow solution is better than no solution. This way, I can fit a 128^3 3D RBC example into a single node on JUWELS booster as opposed to 8 nodes needed when doing spgemm on GPU.
Also, I can fit 256^3 into four nodes compared to 128 nodes before any optimization and I can run 512^3 on 32 nodes. So now a reasonable amount of hardware is enough, which is good!

A better solution would be what they do in Dedalus. When constructing ND matrices, I take Kronecker products of 1D matrices to get very large matrices. In Dedalus, they construct one matrix per dof in one dimension including all dofs in the second dimension. At least when mixing the diagonal Fourier and non-diagonal ultraspherical bases.
Rather than doing one large spgemm, they do many small ones, which prevents spikes in memory consumption.
I think their solution is much better than mine, but also a major overhaul of the code in pySDC would be needed to support this.
I think for the current project this solution should be good enough and it's just a couple of lines..

Note that this PR depends on #569 in order to actually work on GPUs.

[brownbaerchen]

I don't know why the GPU tests fail. They pass when I run them manually.. I expect this is an issue with the testing pipeline rather than with the tests.

BTW, I ran the testing pipeline on JUPITER, so pySDC is now officially compatible with it 🥳