Benchmark metrics collection discussion #13
Description
What do we want to collect
When thinking about tools for running benchmarks, we're thinking a good place to start is to consider what we want to capture with each run. Ultimately, if we're capturing this information, we want it to be useful for decision making on hardware purchases (or anticipating performance) for folks intending to work with parcels
- date of example
- benchmark runtime (float in seconds)
- peak memory consumption (float in GB)
- CPU make/model (string)
- CPU clock frequency ( int in MHz )
- System memory type, size, clock frequency ( e.g. DDR3, DDR4, HBM )
- Disk type and size
- benchmark arguments (nparticles, runtime, dt, chunk size, )
- shasum/some identifier of the input decks (e.g. xarray/uxarray/fieldset hash)
It may also be worth having a separate database that tracks the benchmark descriptions.
How do we want to collect
Ideally using a low overhead, nonintrusive, sampling profiler that does not impact the code that is executed/optimized by the python compiler