[v2] Add support for chunking performance tests, and architectural changes

[aemous]

Issue #, if available:

Description of changes:

• Adds support for evenly chunking performance test cases and only running a specified chunk
• Architectural changes to performance testing
  • Change Summarizer to return metric results in a new Metric data model, which includes additional data like description and unit.

Next big changes to consider:

• low-hanging-fruit: switch binary-content key in begin_iteration to a 'mode' key. Update README as needed.
• low-hanging-fruit: replace our quartile calculation with numpy calls.
• low-hanging-fruit: move all Metric creation into Summarizer
• Instead of returning a list of generators, can we use decorators on function definitions? Each case is a triple (setup, generate_definition, cleanup). Worst case scenario, can just switch to returning those triples.
  • idea: TestCase class with these 3 properties/functions. Can have @performancetest(setup=func, cleanup=func) decorator on a function that accepts iteration as arg and returns the case definition for a single iteration. The decorator would be in charge of wrapping it in a generator to generate/call the function for each iteration, passing the iteration to the function each time. To gather a suite's cases, we get all functions with this decorator. Optional args setup and cleanup will replace begin_iteration and end_iteration with those args.
• Separate consume_case_results and provide_sequence_results to a separate BenchmarkResultHandler component. Put Summarizer as internal object in that component. When we need a benchmark suite with more complex data structure for storing + transforming results, we'd be overriding this component.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.