Refactor, extend benchmarks and incorporate into CI

[glwagner]

This PR refactors the /benchmark directory and incorporates the benchmark scripts into CI. It removes test/benchmark_tests.jl.

Our ultimate objective is to run a curated set of benchmarks regularly, and develop a graphic that tracks the evolution of the benchmarks over time and as performance optimization work is incorporated. This will ensure that the benchmark scripts stay up to date with current API. The benchmark scripts should also be runnable standalone for manual benchmarking. Eventually, we would also like to upload profile artifacts for inspection.

For now, I have moved the existing scripts in benchmarks into an "archive" folder. I think we should delete these, since they will be superceded. However, one question is how/whether we should also incorporate the code in benchmarks/src. I think this code is useful and nice for graphical display, but I also think there is a benefit to having simple benchmarks that consists as single scripts. @ali-ramadhan perhaps you can weigh in here since you developed that code originally.

cc @simone-silvestri @giordano

[ali-ramadhan]

Love this! Most of the code in https://github.com/CliMA/Oceananigans.jl/blob/main/benchmark/src/Benchmarks.jl is to help keep the benchmark scripts short so it'll probably still be useful if we plan on having multiple types of benchmarks.

Some things to think about:

1. There are a lot of benchmarks in https://github.com/CliMA/Oceananigans.jl/tree/main/benchmark Which are still worth benchmarking and which are worth benchmarking regularly?
2. To track the evolution of the benchmarks, will benchmarks be stored somewhere to be compared with future benchmarks?
3. How clean and reproducible do we want the benchmarks to be? Ideally benchmarks will be run on the same machine with the same GPU with no other processes utilizing the CPU significantly or consuming significant amounts of memory. Benchmarking on CI servers like Nautilus may not produce the cleanest benchmarks but maybe if we include the CPU and memory utilization for context then the benchmarks can be interpreted accordingly.

[glwagner]

1. There are a lot of benchmarks in https://github.com/CliMA/Oceananigans.jl/tree/main/benchmark Which are still worth benchmarking and which are worth benchmarking regularly?

I am hoping we can have a system that allows us to benchmark all important situations with just a few scripts which can be run in CI to ensure they do not go stale. Do you think the existing benchmarks (the ones we care about) will incorporate into a new framework?

[simone-silvestri]

We also need to change the pipeline-benchmarks.yaml in some way here. I am not sure how to print out nice artifacts, though.

[glwagner]

@giordano how should we add a new "CanonicalSimulations" or whatever package to the repo?

[giordano]

Could be a project under lib/OceananigansBenchmarks (or whatever you want), with the regular Julia package structure.

[glwagner]

Could be a project under lib/OceananigansBenchmarks (or whatever you want), with the regular Julia package structure.

Thinking something like...

benchmarking/
    example_benchmarking_script.jl
    CanonicalOceananigans/
        src/
        etc...

[giordano]

CanonicalOceananigans/

Only thing is that when you install the package with Pkg, and then you load it, you'd always refer to it by CanonicalOceananigans, which loses the connection with benchmarks.

[glwagner]

CanonicalOceananigans/

Only thing is that when you install the package with Pkg, and then you load it, you'd always refer to it by CanonicalOceananigans, which loses the connection with benchmarks.

Right, trying to scope this... I think when used externally we will mostly use this package to extract the canonical models. I am not sure how much benchmarking tools specifically will be needed in the package