Problem with calculating %ROPE for 2 samples t-test

[sloughon]

Hello,

First of all, I would like to apologize for the triviality of my question, but I am just getting started with the Bayesian approach and this is my first analysis in R.

I was trying to calculate the proportion of samples inside the ROPE. My data is two vectors, one containing the results of group 1, the other containing the results of group 2 (independent samples):

library(BayesFactor)
library(bayestestR)
a=c(value1, value2...)
b=c(value1, value2...)
result <- ttestBF(a,b)
rope(result, range = c(-0.5, inf), ci = 1, ci_method = "HDI", verbose = TRUE)

It gets the correct BF values, however, the % in ROPE are a) do not correspond in any way to the expected value b) are different each time the code is run. I guess I'm making a very basic mistake.

Additionally, I would like to make sure - in such a situation, the ROPE range relates to the size of the effect? I plan to use %ROPE as a measure of how likely an effect that is no greater than small is.

I will be grateful for any help.

[bwiernik]

Can you give a specific real example? The exact values are expected to change slightly from run to run because they are based on simulated values.

[mattansb]

All the bayestestR functions (via insight::get_parameters()) use the raw difference.

If you want the standardized difference (aka Cohen's d), you can:

d <- 0.15
n <- 100

a <- 10 + rnorm(n)  * 30
b <- 10 + rnorm(n, mean = -d) * 30


suppressPackageStartupMessages(library(BayesFactor))
library(bayestestR)

result <- ttestBF(a, b)

result_posts <- posterior(result, iterations = 4e3)
result_posts <- as.data.frame(result_posts)

d_posts <- result_posts[, "delta", drop = FALSE] # delta is the std diff

rope(d_posts, range = c(-0.2, 0.2))
#> # Proportion of samples inside the ROPE [-0.20, 0.20]:
#> 
#> Parameter | inside ROPE
#> -----------------------
#> delta     |     83.98 %

^{Created on 2021-12-21 by the reprex package (v2.0.1)}

---

@strengejacke Does it make sense for insight to give only the raw difference?

[strengejacke]

I thought that for BFBayesFactor objects, get_parameters() returns draws from the posteriors of that object. Isn't this the most straightforward way?

[bwiernik]

It's "scale invariant" only in that the pivot statistic for the mean is scale invariant. Testing an observed standardized difference means that uncertainty in the SD needs to be taken into account (eg, by its own pivot using the noncentrality parameter)

[mattansb]

<tangent 😜>
Hmmm... Isn't that exactly what a t (vs z) test does? Not sure I follow how t/p-values would / should be different between std and raw differences.

I was gonna try an simulate some data to see if the distribution of t values for raw vs standardized is different, but I can't even see how they can be different...

I know that for std-differences, CIs need to account for uncertainty in the standardizing factor, but if a NHST t-test is significant, than 0 is outside of the CI for the std-diff.

set.seed(1)
x <- rnorm(10)
y <- x - 0.7334084

t.test(x, y, var.equal = TRUE)[["p.value"]]
#> [1] 0.0499999

effectsize::cohens_d(x, y) |> print(digits = 7)
#> Cohen's d |                 95% CI
#> ----------------------------------
#> 0.9395614 | [0.0000004, 1.8559572]
#> 
#> - Estimated using pooled SD.

^{Created on 2021-12-24 by the reprex package (v2.0.1)}

[bwiernik]

Yes, I suppose for the hypothesis value of 0, the noncentral T distribution reduces to the central

[mattansb]

Oh no! I conflated "significant testing" with "null hypothesis significant testing"! mea culpa!

[mattansb]

Anyway, in BayesFactor the uncertainty in the SD is accounted for, so the test really is on the standardized scale.

[sloughon]

Thank both of you for fast and very helpful answers! Now I understand why the values ​​were different every time the code was run - because the samples are simulated and why the values ​​were different than expected - because I was building a ROPE interval based on raw differences thinking they were effect size values.

Courtesy of mattansb, I used the following code:

library(BayesFactor)
library(bayestestR)
a=c(value1, value2...)
b=c(value1, value2...)
result <- ttestBF(a, b)
result_posts <- posterior(result, iterations = 4e3)
d_posts <- result_posts[, "delta", drop = FALSE] 
rope(d_posts, range = c(-0.5, Inf),  ci = 1)
# Proportion of samples inside the ROPE [Inf, -0.50]:

Parameter | inside ROPE
-----------------------
delta     |     82.72 %

Now everything works and looks correct. I just wanted to make sure that I can interpret the above result as follows:

"there is 82.72% probability that effect size (in my case it was independent sample t-test so it is Cohen's d) is no lower then -0.5 (i.e., 82.72% of the posterior distribution falls within the ROPE)

[I put my data in the sentence from Kruschke JK, Liddell TM. The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychon Bull Rev. 2018;25(1):178-206. doi:10.3758/s13423-016-1221-4]

Again, I will be grateful for any help.

[bwiernik]

That's reasonable (if you additionally add, "conditional on the data and my prior specification")

[sloughon]

Thanks for all the help

[sloughon]

After reading the above answers and a few attempts on my own analysis, I updated my question (see the post above) - thank you very much for your help and I would be grateful for a look also at it. I hope this post is not spam - delete if necessary.

Again, thank you in advance.