Idea: Make Benchmark.net more suitable for single (long run) benchmarks.

[Danielku15]

I have some benchmark/performance tests which are fairly long-running and are more of a wider integration performance test (aka. "profiling" benchmark) than a the classical "execute often" benchmark. These benchmarks run fairly long and with a variety of parameters.

I know it might be a niche need but nevertheless such tests have their place and value. With the MemoryDiagnoser and EventPipeProfiler configured I get meaningful results and reports out of this benchmarks which I can use to check the general application performance over time. Yes, there might be side-effects and results have to be interpreted more dedicated, but still they can be used for trends (think nightly test rounds).

Due to the nature of these tests being "expensive", I want to ensure every test-case runs exactly once:

1. I setup my test environment and do a custom slight warm-up of the app.
2. Then the main benchmarks run and measure bits like CPU profiling, Memory consumption, allocations etc.

Benchmark.net provides a great foundation for this and it allows me to keep those kind of benchmark close to the more traditional ones which are show-running & often repeated executions to measure the performance of specific logic.

---

Now to the main problem in the current library behaviors: Despite my config of [SimpleJob(RunStrategy.ColdStart, warmupCount: 0, launchCount: 1, iterationCount: 1)] every benchmark gets executed 4 (!) times.

1. The first execution is part of the normal iteration via Engine.Run -> RunIteration -> WorkloadAction
2. The second execution is via Engine.Run -> Engine.GetExtraStats -> Engine.MeasureWithGC -> WorkloadAction.
3. Additionally, the host seem to execute the job process with the same parameters twice. Once for the normal results, and once for the "Diagnostic run"

---

Unless I miss some configuration I could use, it would be great if Benchmark.net could honor such single-execution setups in some way:

1. The lib should not execute an additional run of the benchmark to collect the GC details. Maybe it can simply use the "last" iteration to collect the extra stats like GC allocations?
2. The lib should not execute a separate diagnostic run of the benchmark doubling things again. Similar to the previous proposal it could use the main execution.

Both new features would obviously require special configuration as users need to be aware of the impact (e.g. profiler overhead etc). Still I think they would be a valuable addition to maintain some general application performance related tests using Benchmark.net.

Not using BDN for such tests means: users lose a variety of nice features for doing repetitive application performance testing (parameterization, reports, profiling,..) but they also they need to be maintained "somewhere else" compared to the traditional benchmarks.

I'm aware that such requests have been stated in the past, but typically they were typically closed with "you are doing benchmarking, you should execute multiple times". That's why I tried to give deeper insight in why this would make sense to have.

Let me know what you think.

[timcassell]

Maybe something like RunStrategy.Diagnostic where it skips the main run and only runs the extra iteration? I'm wary of such a strategy, because GC results will be unreliable. Even with your current setup, the GC results are measuring tier 0 code-gen, which maybe you're okay with, but with a strategy of measuring a single invocation, it will also be measuring any allocations produced by the first JIT and any static constructors that your benchmark triggers.

[timcassell]

Another option is we could allow launchCount: 0 so you can skip the first launch and just run the diagnostic launch, reducing it from 4 to 2 invocations. [Edit] But this idea won't really work for NoOverhead diagnosers, they would also be skipped.

[Danielku15]

Maybe something like RunStrategy.Diagnostic where it skips the main run and only runs the extra iteration?
Another option is we could allow launchCount: 0 so you can skip the first launch and just run the diagnostic launch, reducing it from 4 to 2 invocations.

As long as the base criteria is fulfilled, I'd be fine with any approach:

1. Report data should still be collected.
2. GC information should be collected
3. All diagnosers should run

As far I could see, the current behavior is:

1. The GetExtraStats execution does not collect any measurements.
2. The second launch only happens if you have configured a diagnoser which sets RunMode.ExtraRun. I have variants with and without EventPipeProfiler.

This might have an impact on how things need to be implemented. I'm fine with providing additional configurations as needed. 😊

Even with your current setup, the GC results are measuring tier 0 code-gen, which maybe you're okay with, but with a strategy of measuring a single invocation, it will also be measuring any allocations produced by the first JIT and any static constructors that your benchmark triggers.

Yes, generally you are right. While its not 100% deterministic, I try to address these aspects by running a "custom warmup" in GlobalSetup where I also setup the application itself. This moves critial parts outside the collection time window. This way most parts should be initialized and jitting might happened for some hot paths already.

[timcassell]

@AndreyAkinshin What do you think?