Default CI level back to 95%? 😱

[DominiqueMakowski]

TLDR for new readers

Currently, most of bayestestR's functions have ci=0.89 by default, yielding 89% CIs. We are thinking of changing that (maybe to 95%). To avoid any surprises or breaks when you update the package, please set the ci argument explicitly. For example, to retain the current behaviour, just add ci=0.89 to most of bayestestR's functions (or in functions related to it, such as parameters::model_parameters()).

Discussion

Might be throwing a stone in the water here, but it's good to rechallenge our certainties :)

When we switched from 95% to 89%, the main reason was that there was no objective reason to keep 95% as it is a purely arbitrary value. And 89%, suggested by McElreath, has at least for it to be a prime number (not that it changes anything, but it might be nice [for some strange people such as @strengejacke 😁]). The real underlying motivation was also to shake the common (and mindless) procedures, to make people actually think about what they are doing.

Now, I think I have some new arguments to throw in the pot.

• 89% is aesthetically ugly. Don't kill the messenger here, but it's true (see this, or Eilers, 2004). We tend to prefer round values (and having an aesthetically pleasing world is very important). For regular folks as me, 89 has a particular taste of unfinished business.
• 95% can facilitate bridges with the Frequentist framework. In times of reproducibility crisis, it doesn't help to change all the references. Keeping 95% would at least allow for some level of comparison with frequentist studies (as credible and confidence intervals are in the end often close), and would also facilitate replication attempts and checks. Note that the downside of this argument is that it creates a meta-stable system in which moving away from bad procedures is very complicated (because by following this argument, one could argue to keep using old methods for continuation and comparison purposes).
• 89 is too small? IMO one of the goals of credible intervals is to address the fact that many distributions have virtually no ends. For instance, in a normal distribution of mean 0 and SD 0.1, a value of 10^99 is still (although infinitesimally) possible. Thus, the idea was to take a range of the most plausible values, mainly for convenience, to say "this distribution goes from here to here". Hence the approximation we learn in class that a normal distribution has a range of 6 SD, because it makes some practical sense. Within this context, 89 appears as being a bit too narrow as a summary of an (at least normal) distribution, cutting the tails quite heavily.
• Relationship with SD. If we take our beloved normal distribution (which is the most "common" (or aspecific) distribution), 95% is actually an approximation in the so-called 68, 95, 99.7 rule. 95% is the approximation of 0.9544997, which is the range of the population contained within 2 SDs on each side of the mean. Although this 95% value likely originated precisely from this relationship with the SD in the frequentist framework (allowing for some nice equation simplifications), the reason to use it in a Bayesian framework would be to connect the CI to the SD (instead of reporting both, which I never liked), which is also a familiar and graspable index.

All opinions are appreciated

# Code for CDF
x <- bayestestR::distribution_normal(10^8)
length(x[x <= 2 & x >= -2]) / 10^8
0.9544997

[strengejacke]

But some of the coolest papers use 89% HDI... ;-)

89% is aesthetically ugly

[strengejacke]

Of course, there is no reason against 95% (as there is also no reason against 89%). Computational stability is an argument from the Stan developers (and Kruschke) to use 90%, so where to go now? Is 89 too narrow? is 95 to imprecise? is 90 ok (but if we break conventions, why not stick to 89?)?

I'm not dogmatic here and open for any changes. One reason against 95% is, and I will cite (and translate) a German song text:

We do not have to do things as we used to do, just because we used to do it as we used to do it.

(not sure if this translation really works...?)

[mattansb]

I thought some of the motivation was Kruschke's idea that for a 95% CI you need a lot of data, and so it would be no more than 90%?

[DominiqueMakowski]

"Computational stability" is indeed one of the strong argument for lowering the percentage. But I'd be curious to see some figures showing the impact of the CI level on the results... does any know where to find that?

[strengejacke]

does any know where to find that?

What about those books or papers where this claim stems from?

[strengejacke]

I don't understand everything, but I think this post at least contributes a bit to your question...

https://betanalpha.github.io/assets/case_studies/markov_chain_monte_carlo.html

[DominiqueMakowski]

I think I have become pro 95 tho 😅 but yeah I guess we've gone too far with 89 to make such a breaking change

[strengejacke]

Let's wait for the 2nd edition of Statistical Rethinking (should be in my office right now...) and if there's still the 89%, we will close this, else we'll make a breaking change to 95.

[strengejacke]

Ok, in the current (2nd) edition of Statistical Rethinking, it seems like McElreath still advices against 95% CI (just to avoid falling into the trap of thinking of NHST when you hear "95%"), but he doesn't seem to say 89% is the number. So... We could

a) stick to 89%
b) change to 90% (like rstanarm)
c) change to 95% (like brms and frq)

should we run runif() to decide? I prefer a) or b).

[strengejacke]

and the winner is...

table(sample(c("a", "b", "c"), 1000, replace = TRUE))
#> 
#>   a   b   c 
#> 343 331 326

^{Created on 2020-05-11 by the reprex package (v0.3.0)}

[mattansb]

lol

I also prefer a or b (b a bit more).

[strengejacke]

I'm open to either a) or b)
@DominiqueMakowski you're the maintainer...

[DominiqueMakowski]

the time that has passed has strongly reinforced my preference for c 😅 it seems like we are in a pickle...

The argument of not doing something just because it reminds of something not good is a bit weak imho. And au contraire you could argue that 95% is great because it allows for comparison with previous results and, you know, reproducibility and all that kind of jazz. But aside from that, the real game changer for me is the correspondence between the 95% CI and the SD (or the SE - right @strengejacke :p)...

[DominiqueMakowski]

If being maintainer doubles my vote weight, we're still stuck between 2c vs. 2b 😁

also, I do remember you were more a brms than a rstanarm guy @strengejacke 🤔

[mattansb]

What is the correspondence between 95% CI and SD?

(And a reminder: the SD of the posterior is not the SE 😋)

[strengejacke]

So this means that every year or so we should switch between 0.89 and 0.95?

😄

we should default ci = runif(1)...

[strengejacke]

Maybe we can start by adding a startup message saying that it will change in the future version?

Maybe we can start by adding a startup message saying that it may change in the future version?

[mattansb]

👆 definitely this!
"Default CI width may change in the future, so it's safest to specify them, even if equivalent to the defaults."

[strengejacke]

we may tag Richard McElreath, just to get another unbiased opinion here...? 😇

[DominiqueMakowski]

we may tag Richard McElreath, just to get another unbiased opinion here...? 😇

tho I reckon it might be a slightly biased opinion 😅

[DominiqueMakowski]

I think we should go ahead with this for next release to finally remove this ugly on-attach message 😁

[mattansb]

So this means that every year or so we should switch between 0.89 and 0.95?

[DominiqueMakowski]

For the record, see here for some data related to the stability issue

[botondbalogh]

Dear All, I am a user of your R packages. Thanks for all your contributions, they are super helpful!!!
And thank you for having open discussions about such issues as this one.

I would argue to keep the 89% default CI level for educational purposes.

To explain, I have scientific background (plant pathology, agriculture), but limited formal education in statistics. I had no idea that R existed until about 5 years ago; and have never heard of Bayesian statistics until a bit over a year ago. Now ,I use rstanarm together with bayestestR and modelbased as a replacement for the agricultural-industry standard ANOVA + LSD tests. If you had not had 89% as the default, I would have never researched the topic to realize that the 95% is not set in stone.
My point is: If you are an expert in this field you know to consider the CI level, so the default is not really limiting you. But amateurs, like myself, will only give a thought to the defaults that look weird. This 89 is surely an "ugly" number that will stop one on its track. If you change it to 95, the next "me" will just accept it without a second thought, and will never know that CI level is something to consider.

[DominiqueMakowski]

Thanks for your feedback ☺️

It's true that for now 89 is good as a "habit disruptor", but IMO it's not so much a property of the number per se, but rather of the fact that it's extraordinary (in the primary sense, i.e., it's rare and stands out). But accepting this value as a default goes exactly against the philosophical stance by which it was created (which is not to use defaults and to think about what to chose). So there is something oddly antithetical about the fact of cementing 89 as a default choice. On top of this philosophical conflict, the fact is that if all people and software adopt 89, it will become just another convention... and again will a generation of rebels will rise, cheering for the instauration of 95. Maybe it's inevitable though... as Marxists say, the history of mankind is a dialectical process...

One option (supported for instance by @mattansb) is to simply remove the default and force users to chose. But that's IMO a bit a bit too authoritarian and it makes the assumption that new users have the time / willingness / comfort to spare some time reading and learning about the issue, which is not the most inclusive in that regards. We could argue that they should take some time to learn about it, since it's an important issue (especially given that in the bayestestR's documentation we tried to make that issue as salient and accessible as possible - so "I don't have time / resources" is hardly a valid excuse), but still.

I think the debate-breaker is that, all things considered, 95 is not that bad of a convention, and it has some arguments for it. I'd be very keen on changing the convention or even forcing people to chose if it was really bad and if the alternatives were a lot better, but it does not seem to be the case.

Now, I see changing this convention more as a pseudo-iconoclasm (pseudo as it changes one convention for another), draped under the cloak of scientific revolution.

[strengejacke]

There is still no consensus on the question which CI-level to use. I think the latest update of bayestestR reverted the 89% CI and now defaults to 95% again. The dark side has won, but the return of the Jedi is possibly happening soon. ;-) At least, there's a new hope, namely the parameters package, which still uses the 89% CI for Bayesian models. But I also think this debate is still ongoing, and not closed.

[DominiqueMakowski]

The dark side has won, but the return of the Jedi is possibly happening soon

Another reason for people to use report which follows the true way of the force, 95, mwahahaha

[bwiernik]

Nobody hates p<.05 more than me (my doktorgroßvaters are Jack Hunter and Frank Schmidt), and if I had to choose a default, I'd pick 90% or 80% (5 or 10% above and below), but I think this convention is long cast, and there is very little value in messing with a default that folks can generally adopt reasonable inferences around.

[willizhang]

Since they are used to summarise posterior distribution, how about we report all: 89%, 90%, 95% credible intervals? This can be used for comparison purpose and also deliver the message that the cut-off is arbitrary?🤔

[strengejacke]

we already support multiple values for the ci argument, but I think the default should be just one value.

[IndrajeetPatil]

E.g. see here

[DominiqueMakowski]

Yes the idea is that a descriptive table can only be so much descriptive. Even if we returned 10 different levels of CIs in a table, it would be just hard to convey the sense of uncertainty. The best alternative is definitly to plot the posteriors

[willizhang]

Many thanks for the replies. They are really helpful. I am thinking about including posterior distribution (the above plot looks beautiful!) in the supplementary file while report one interval in the main manuscript.

[bwiernik]

This sort of parameters plot can also be really useful as a main text figure

[pglpm]

The 89% credibility interval has this special property: knowing whether the true value is within or without this interval, amounts to almost exactly 0.5 shannons of uncertainty; 0.499916 Sh to be more exact. So that's about "half a binary uncertanity", so to speak, on the log-scale of Shannon information. Or almost exactly half the uncertainty of a 50% credibility interval.

The 90% credibility interval amounts to 0.468996 Sh, and the 88% interval to 0.557438 Sh; both a bit farther away from 0.5. The interval corresponding to 0.5 Sh is 88.997%. Finally, the 95% interval corresponds to 0.286397 Sh.

Just a curiosity, not an argument in favour of 89% :)