Investigate slowdown due to horizontal diffusion (AMD and Smagorinsky)

[petebachant]

Preliminary results from @trontrytel:

[trontrytel]

I should just add that I was using AMD parameterization for horizontal diffusion, which is pretty involved and probably not what we want to spend time optimizing. I'll add constant diffusion as an option and run another test. Will update the plot

[petebachant]

Thanks! Can you point to what configs were used for each so I can profile them?

[trontrytel]

#
# Basic
#
FLOAT_TYPE: "Float32"
dt: "30secs"
t_start: "0secs"
t_end: "3days"
start_date: "20100101"
x_max: 300000.0
x_elem: 6
y_max: 300000.0
y_elem: 6
z_max: 30000.0
z_elem: 43
z_stretch: true
dz_bottom: 30.0
#h_elem: 60
h_elem: 16
nh_poly: 3
config: "sphere"
initial_condition: "MoistBaroclinicWave"
perturb_initstate: true
detect_restart_file: false
restart_file: ~
reference_job_id: ~
log_progress: true
log_to_file: false
use_itime: false
device: "auto"
toml:
  - "toml/longrun_aquaplanet_1M.toml"
output_dir: "/glade/derecho/scratch/ajaruga/scaling_1M_dt_30s_dEDMF_max_newton_iters_ode_3_h_elem16_1GPU/"
#
# Numerics
#
ode_algo: "ARS343"
tracer_upwinding: "vanleer_limiter"
energy_upwinding: "vanleer_limiter"
apply_limiter: false
discrete_hydrostatic_balance: false
deep_atmosphere: true
split_ode: true
rayleigh_sponge: true
viscous_sponge: false
approximate_linear_solve_iters: 2
max_newton_iters_ode: 1
use_newton_rtol: false
newton_rtol: 1.0e-5
use_auto_jacobian: false
auto_jacobian_padding_bands: ~
use_dense_jacobian: false
update_jacobian_every: "solve"
jvp_step_adjustment: 1.0
debug_jacobian: false
use_krylov_method: false
use_dynamic_krylov_rtol: false
krylov_rtol: 0.1
divergence_damping_factor: 5.0
eisenstat_walker_forcing_alpha: 2.0
vorticity_hyperdiffusion_coefficient: 0.1857
scalar_hyperdiffusion_coefficient: 0.929738
#
# Tests we don't do
#
check_steady_state: false
test_dycore_consistency: false
bubble: false
advection_test: false
check_conservation: false
reproducibility_test: false
check_nan_every: 1024
perf_summary: false
#
# SLEVE
#
mesh_warp_type: "SLEVE"
sleve_s: 10.0
sleve_eta: 0.7
#
# Topography
#
topography: "Hughes2023"
topo_smoothing: false
topography_damping_factor: 5
#
# Surface properties
#
albedo_model: "ConstantAlbedo"
surface_temperature: "ZonallySymmetric"
surface_setup: "DefaultExchangeCoefficients"
prognostic_surface: "SlabOceanSST"
surface_thermo_state_type: "GCMSurfaceThermoState"
add_isothermal_boundary_layer: true
insolation: "idealized"
disable_surface_flux_tendency: true
#
# Gravity wave
#
non_orographic_gravity_wave: false
dt_nogw: "1800secs"
orographic_gravity_wave: ~
#
# Radiation
#
rad: ~
dt_rad: "6hours"
radiation_reset_rng_seed: false
aerosol_radiation: false
prescribe_clouds_in_radiation: false
#
# Clouds, aerosol and tracers
#
moist: "nonequil"
precip_model: "1M"
cloud_model: "grid_scale"
dt_cloud_fraction: "3hours"
call_cloud_diagnostics_per_stage: false
implicit_noneq_cloud_formation: false
prescribed_aerosols: []
idealized_h2o: false
idealized_clouds: false
prescribe_ozone: false
co2_model: ~
#
# SGS model
#
vert_diff: false
# vert_diff: "DecayWithHeightDiffusion"
# vert_diff: "VerticalDiffusion"
smagorinsky_lilly: false
amd_les: false
implicit_diffusion: false
hyperdiff: "CAM_SE"
#
# EDMF
#
turbconv: ~
energy_q_tot_upwinding: "vanleer_limiter"
tracer_upwinding: "first_order"
edmfx_mse_q_tot_upwinding: "first_order"
edmfx_tracer_upwinding: "first_order"
prognostic_tke: true
edmfx_detr_model: "Generalized"
edmfx_entr_model: "PiGroups"
edmfx_nh_pressure: true
edmfx_sgs_mass_flux: true
edmfx_sgs_diffusive_flux: true
#
# CF sites and other we don't use
#
cfsite_number: "site23"
site_latitude: 17.0
site_longitude: -149.0
external_forcing: ~
external_forcing_file: ~
#
# Output
#
output_default_diagnostics: false
dt_save_state_to_disk: "30days"
netcdf_interpolation_num_points:
  - 1440
  - 720
  - 43
diagnostics:
  - short_name:
      - "rhoa"
      - "ua"
      - "va"
      - "wa"
      - "ta"
      - "hus"
      - "clw"
      - "cli"
      - "hussn"
      - "husra"
    period: "60days"
    reduction: "inst"

[trontrytel]

When running with AMD diffusion I was also commenting out the vertical component in this line:

ClimaAtmos.jl/src/prognostic_equations/remaining_tendency.jl

Line 358 in a85ba64

vertical_amd_tendency!(Yₜ, Y, p, t, amd)

[petebachant]

@trontrytel how did the constant horizontal diffusion compare?

[petebachant]

I just ran a quick benchmark with Smagorinsky and it was a ~10% slowdown, so not as bad as AMD I guess.

[trontrytel]

It was faster! But something else changed about the model and I can't place neatly with other plots without having to re-do all of them :/

I would vote to use the constant horizontal diffusion for benchmarking. Or no diffusion? cc @szy21 ?

[szy21]

Yes, we can use the constant horizontal diffusion.

[petebachant]

Found what appears to be a small opportunity for a speedup in #4182, but overall the constant horizontal diffusion is not nearly as heavy as Smagorinsky (~1--2% versus ~10%). Haven't tried AMD yet. Is the thought that you'll run with const horiz for now to stabilize the simulation, or are these other diffusion models important to optimize?

[szy21]

I think we will use constant horizontal diffusion for now. Smagorinsky and AMD will be used for configs other than AMIP, and if there are obvious things to optimize, it would still be good to do that. But if not, we can focus on other parts of the code now.