GPU compilation error using 1M microphysics

[nefrathenrici]

Some configurations with 1M result in a compilation error on GPU.

Example:

ERROR: LoadError: InvalidIRError: compiling MethodInstance for ClimaCoreCUDAExt.copyto_stencil_kernel!(::Field{…}, ::Base.Broadcast.Broadcasted{…}, ::ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{…}, ::NTuple{…}, ::ClimaCore.DataLayouts.UniversalSize{…}, ::ClimaCore.DataLayouts.NoMask, ::CartesianIndices{…}) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to *)
Stacktrace:
 [1] *
   @ ./operators.jl:596
 [2] compute_Δ𝛼_linvanleer
   @ /scratch/clima/slurm-buildkite/climacoupler-longruns/948/depot/default/packages/ClimaCore/y7ZbY/src/Operators/finitedifference.jl:1498
 [3] multiple call sites
   @ unknown:0

[szy21]

Adding to this, I got this when running it on central: https://buildkite.com/clima/climaatmos-ci/builds/26356#0199ccaa-86ee-4416-b2f4-3e73486c0db6/186-243

ERROR: LoadError: Kernel invocation uses too much parameter memory.
--
  | 5.961 KiB exceeds the 4.000 KiB limit imposed by sm_60 / PTX v8.8.

[trontrytel]

Those are two different errors :(

[szy21]

Yeah, I think clima / derecho has a larger parameter memory, but I wonder if we should reduce the memory somehow...

[szy21]

(otherwise it would be hard to add a ci job for gpu 1m, so we can't track it)

[trontrytel]

For the diagnostic EDMF issue I narrowed it down to those lines:

ClimaAtmos.jl/src/prognostic_equations/edmfx_sgs_flux.jl

Lines 293 to 302 in 47166b3

	@. ᶜa_scalar =
	ᶜχʲ * min(
	min(draft_area(ᶜρaʲs.:($$j), ᶜρʲs.:($$j)), a_max),
	FT(0.02) / max(
	Geometry.WVector(
	ᶜinterp(ᶠu³_diff),
	).components.data.:1,
	eps(FT),
	),
	)

I don't fully understand why, because we have a very similar pattern above for q_tot. But maybe looping over all tracers adds some complexity to the compiler?

cc @nefrathenrici , @dennisYatunin

[szy21]

I remember Dennis mentioned the for loop may not work on gpu. So maybe try changing it to unrolled_foreach?

[trontrytel]

It's not the above.

Apparently that particular configuration was trying to run with tracer_upwinding = vanleer_limiter I don't understand why, but this indeed does not compile on a GPU. However if you switch it to tracer_upwidning = first_order the code compiles just fine.

[trontrytel]

I propose to switch to first_order by default for now and move on to actually looking at AMIP results. Unless someone with more ClimaCore experience can give me a hint here

[szy21]

Wow, strange. I think Sajjad is going to change the diagnostic mass flux back to what we have before, and maybe there will be no issue. Does it solve the prognostic EDMF issue as well?

In any case, I think it would be good to start looking at AMIP, so I think we can switch to first_order. And I think it may be ok if we only change it for microphyscis tracer (not e_tot and q_tot). We should definitely look into why vanleer doesn't work though. @dennisYatunin Could someone in the software team look into this?

[trontrytel]

I talked with Dennis some more and I have a version that is/should be working. We need to fix something in ClimaCore or hard code more 1 instead of j in the loop. I'm ooo tomorrow, but I'll have a PR by Wednesaday and an estimate on how long diag EDMF + 1M runs and how bad it is.

It will not solve the out of memory issue you saw in prognostic EDMF. I don't think they are related.

[dennisYatunin]

ClimaCore bugfix is in CliMA/ClimaCore.jl#2377

[trontrytel]

I spend some more time with it, and I really don't understand why vanleer_limiter is not working for microphysics tracers. In the interest of time I'll add a separate flag and set it for first_order for now. Once we decide it's really needed to make it work with vanleer I can spend more time on it