Slow get_expected_radiance() with MPAS (redundant calls to find cellID?)

[lkugler]

Hi,
We found that get_expected_radiance() in obs_def_rttov_mod.f90 is slow with MPAS but not WRF.
In my tests, the model_interpolate() block L3603 - L3920 takes ~80% of the walltime per observation (0.56-0.94s) while the call to rttov_direct() (the RTTOV function) is < 0.001 sec per obs.

DART/observations/forward_operators/obs_def_rttov13_mod.f90

Line 3603 in 146c853

GETLEVELDATA : do i = 1,numlevels

Can it be that the search is performed again for every variable and vertical level that is requested?
At least in WRF and MPAS, the cell index is the same for all vertical layers and variables, so it only needs to be computed once for e.g. 20 variables x 100 layers.

DART/models/mpas_atm/model_mod.f90

Line 6388 in 146c853

! do this exactly once.

Could you take a look at this? Thanks a lot!
Lukas

No issue in WRF

For comparison, I timed perfect_model_obs() with MPAS on derecho. 200 obs took 3 minutes and the interpolation block (= bottleneck) took 0.81 (+/- 0.005) seconds per observation (each obs at the same location for simplicity). On the Vienna cluster, PMO with WRF and 200 obs took 11s. Single process in both cases, but WRF and MPAS were different domains.

Error Message

None.

Which model(s) are you working with?

MPAS_atm

Version of DART

I compared (freshly compiled) v10.1.0 and v11.17.0.
For 200 observations, timings are 3:55 and 3:25 respectively (min:sec) on derecho.

Have you modified the DART code?

No

[hkershaw-brown]

Thanks for reporting this Lukas. The mpas model_mod is using a search to find the nearest cell, where as wrf it is just a calculation.

Like you say, this search is unnecessarily repeated a lot (e.g. 20 variables x 100 layers) all call call xyz_find_nearest(cc_gc, pointloc, cell_locs, closest_cell, rc) to get the same 'closest_cell'

DART has the interface "model_interpolate (single qty, single location)" when really the more complicated forward operators want "column_interpolate(several qtys, single horizontal location, whole column)" - rttov, chemistry fwd ops (#912).

I think if you want a quick solution to run faster you could save lon, lat, cellid, in model_interoplate, and if the incoming location equals the last location, use the saved value of cellid rather than redoing the search cell_ok_to_interpolate -> find_closest_cell_center

DART/models/mpas_atm/model_mod.f90

Line 1179 in 146c853

cellid = cell_ok_to_interpolate(location)

note for someone looking at this, xyz location mod is always doing an exhaustive search in get_close
#971 (comment)

[nancycollins]

maybe i'm confused, but that code (exhaustive search) is in the threed_cartesian location mod. mpas is using the threed_sphere locations and also the xyz location mod. the xyz location mod divides space up into boxes and then does a search through only those locations in boxes close enough to be candidates. it may not have great defaults - someone could increase nx, ny, nz in the namelist and see if that speeds things up.

[hkershaw-brown]

maybe i'm confused, but that code (exhaustive search) is in the threed_cartesian location mod. mpas is using the threed_sphere locations and also the xyz location mod. the xyz location mod divides space up into boxes and then does a search through only those locations in boxes close enough to be candidates. it may not have great defaults - someone could increase nx, ny, nz in the namelist and see if that speeds things up.

Thanks Nancy, this is just me being sloppy in my comments, just wanted to make sure anyone doing a deep dive into location mods has a look at the other issues.

[nancycollins]

good to make a note to look at this in the threed_cartesian module. at the time it was written i think only the CM1 model was using it, and running with idealized cases. so not a huge numbers of obs and not huge grids. that would make it easy to miss that it's doing something inefficient.