Tracking diagnostics compute / write times in ClimaAtmos #4043
Description
Simulation time: Baroclinic Wave (Non-equilbrium, 1M Microphysics) for 72h, on (1,2,4) gpus.
Device: NVIDIA-H100
ClimaAtmos v0.31.5
Diagnostic frequency:
Default DIAG_DEF with 1d averages
instantaneous DIAG_INST (1d outputs)
both default and instantaneous(DIAG_ALL)
and no diagnostics NO_DIAG
| nGPUs | Diag | Log Filename | Walltime (s) | Walltime/Timestep (s) | SYPD |
|---|---|---|---|---|---|
| n1 | DIAG_ALL | n1_DIAG_ALL_err-393.txt | 264.37 | 0.203000 | 2.686 |
| n1 | DIAG_DEF | n1_DIAG_DEF_err-389.txt | 221.90 | 0.171000 | 3.200 |
| n1 | DIAG_INST | n1_DIAG_INST_err-382.txt | 222.20 | 0.171000 | 3.196 |
| n1 | NODIAG | n1_NODIAG_err-385.txt | 158.86 | 0.122000 | 4.470 |
| n2 | DIAG_ALL | n2_DIAG_ALL_err-394.txt | 202.40 | 0.156000 | 3.509 |
| n2 | DIAG_DEF | n2_DIAG_DEF_err-390.txt | 156.97 | 0.121000 | 4.524 |
| n2 | DIAG_INST | n2_DIAG_INST_err-383.txt | 155.97 | 0.120000 | 4.553 |
| n2 | NODIAG | n2_NODIAG_err-386.txt | 94.48 | 0.072000 | 7.516 |
| n4 | DIAG_ALL | n4_DIAG_ALL_err-395.txt | 166.44 | 0.128000 | 4.267 |
| n4 | DIAG_DEF | n4_DIAG_DEF_err-391.txt | 165.53 | 0.127000 | 4.290 |
| n4 | DIAG_INST | n4_DIAG_INST_err-384.txt | 120.39 | 0.092000 | 5.899 |
| n4 | NODIAG | n4_NODIAG_err-387.txt | 58.92 | 0.045000 | 12.052 |
Diagnostics schedules are defined here: https://github.com/CliMA/ClimaAtmos.jl/blob/main/src/diagnostics/default_diagnostics.jl