The metrics for the mean and standard deviation of (percentage-wise) GC are not consistent with each other

[frankwswang]

I was trying to understand better how each number in the printed information of a BenchmarkTools.Trial is derived. I noticed an inconsistency between the mean and the standard deviation (SD) computed for the ratioed GC time. Consider the following case:

julia> using BenchmarkTools

julia> using Statistics

julia> function foo(a::Int)
           v = rand(3,3)
           (v.^a) |> sum
       end
foo (generic function with 1 method)

julia> b = @benchmark foo(2)
BenchmarkTools.Trial: 10000 samples with 978 evaluations per sample.
 Range (min … max):  69.632 ns …  1.398 μs  ┊ GC (min … max): 0.00% … 87.32%
 Time  (median):     71.268 ns              ┊ GC (median):    0.00%
 Time  (mean ± σ):   78.708 ns ± 44.711 ns  ┊ GC (mean ± σ):  6.78% ± 10.74%

  █▄                                                          ▁
  ███▆▅▅▅▄▄▅▄▆▇▅▄▅▇▆▃▄▃▁▁▄▁▁▁▁▃▁▁▁▁▆▆▄▆▆▃▄▁▄▄▁▁▄▃▁▁▁▁▁▁▁▄▆▇██ █
  69.6 ns      Histogram: log(frequency) by time       315 ns <

 Memory estimate: 288 bytes, allocs estimate: 4.

The SD is computed w.r.t. the ratios of GC time over the total time for each sample:

julia> isapprox(0.1074, Statistics.std(b.gctimes ./ b.times), atol=1e-4)
true

which can also be verified through the source code.

However, for the mean, instead of the Mean of the Ratios (MoR), the printed information showed the Ratio of the Means (RoM):

julia> isapprox(0.0678, Statistics.mean(b.gctimes ./ b.times), atol=1e-4)
false

julia> isapprox(0.0678, Statistics.mean(b.gctimes) / Statistics.mean(b.times), atol=1e-4)
true

The corresponding source code verification can be found here.

Both MoR and RoM have statistical significance. However, if we want to print the mean and SD for the GC time together, like BenchmarkTools.jl's current printing format, I think they should be consistently set w.r.t. the same random variable: the ratioed GC time (GC time / total time per sample).