Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ratio value is not 1.00 for baseline #2648

Open
jjanuszkiewicz opened this issue Sep 25, 2024 · 2 comments
Open

Ratio value is not 1.00 for baseline #2648

jjanuszkiewicz opened this issue Sep 25, 2024 · 2 comments

Comments

@jjanuszkiewicz
Copy link

jjanuszkiewicz commented Sep 25, 2024

For starters, please excuse me if this is not a bug, but my misunderstanding of what the Ratio value represents.

I've created 2 benchmarks, one of them is marked as baseline. I'm getting results such as:

| Method        | Mean       | Error     | StdDev    | Median   | Min      | Max        | Ratio | RatioSD | Baseline | Allocated | Alloc Ratio |
|-------------- |-----------:|----------:|----------:|---------:|---------:|-----------:|------:|--------:|--------- |----------:|------------:|
| SomeMethod_V1 | 1,161.1 ms | 319.86 ms | 943.10 ms | 739.0 ms | 524.7 ms | 5,106.1 ms |  1.40 |    1.35 | Yes      |  12.68 MB |        1.00 |
| SomeMethod_V2 |   498.8 ms |  18.31 ms |  53.99 ms | 469.1 ms | 442.1 ms |   684.2 ms |  0.60 |    0.25 | No       |  10.04 MB |        0.79 |

I expected Ratio to be 1.00 for SomeMethod_V1, which is the baseline (it is 1.00 for Alloc Ratio).

My benchmark code:

[MemoryDiagnoser]
[SimpleJob(RunStrategy.Monitoring, warmupCount: 1, iterationCount: 100)]
[MinColumn, MaxColumn, MeanColumn, MedianColumn, BaselineColumn]
public class SomeMethodBenchmarks
{
	[GlobalSetup]
	public async Task GlobalSetup() { /* ... */ }

	[GlobalCleanup]
	public void GlobalCleanup() { /* ... */ }

	[Benchmark(Baseline = true)]
	public async Task SomeMethod_V1() { /* ... */ }

	[Benchmark]
	public async Task SomeMethod_V2() { /* ... */ }
}

I'm using BenchmarkDotNet 0.14.0 on .NET 8.0.8.

Am I doing something wrong, or is this a bug?

@crone66
Copy link

crone66 commented Sep 25, 2024

I noticed the same thing for some of my benchmarks.

@eNeRGy164
Copy link

I got one as well that surprised me a bit, not a big difference though.
Latest nightly, ratio 1.001

Screenshot 2024-09-26 213336

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants