From d48da2277941e9de2b87ad750f5201d6aae55993 Mon Sep 17 00:00:00 2001 From: Charles Blake Date: Fri, 3 May 2024 10:34:58 -0400 Subject: [PATCH] Add note about improved experimental environ improving distro hostility and re-structure concluding paragraph to be more direct & ref [8] again. --- doc/tim.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/doc/tim.md b/doc/tim.md index d09eb84..161397c 100644 --- a/doc/tim.md +++ b/doc/tim.md @@ -163,11 +163,14 @@ study showed the situation is mostly very leptokurtotic.. (I saw excess kurtosis over 12) meaning wild tail events are much more common than expectations from light-tailed noise. So, one cannot use sigma alone for t-tests. This wild distribution itself is likely irreproducible over time or across test machines. - -So, we can answer the question "Does it work?" with "kinda, but take any A/B -t-tests with a cube of salt". Deviations beyond 10 sigma with no underlying -difference are far too common, but errors are still small in absolute terms and -so can still separate fairly subtle effects. +Running that set of tests with fixed CPU frequency and very minimal background +activity would very likely make these distributions less hostile. + +So, we can answer the question "Does it work?" with "kinda!". 10σ devs with no +underlying difference are far too common, yet errors are still small in absolute +terms letting you separate fairly subtle effects. So, it seems useful as long +as you take any A/B pseudo-t-tests with a "cube of salt" a bit bigger than the +one common in particle physics.[^8] Other issues ============