evidence 'authority' ?

[cordphelps]

I'm looking for tools to examine the authority of the 'evidence' used in posterior derivations. Is there a concept similar to frequentist 'power' for determining appropriate sample size for bayesian evidence? Thank you.

[mattansb]

Power in the frequentist sense isn't really a thing - not common to mix error rates into Bayesian stats. There also isn't as much of a problem with sequential analysis, so the general idea is - collect data until you have enough evidence.

Having said that - some thoughts:

1. Use a Bayesian simulation study - specify priors, sample from their (prior) predictive distribution and see how many observations you need to find what you are looking for.
2. Maybe take a look at Bayes factor design analysis https://github.com/nicebread/BFDA

[cordphelps]

Power is not my end game. I'm looking for appropriate sample size (and a way to justify it). I only refer to power because I've been conditioned by Cohen to calculate sample size based on effect size and power.

I'm working with count data (numbers are less than 7 and there are many zeros). I suspect that a posterior based on one evidence observation would not be interesting, but I can't see how to determine 'enough' evidence.

[mattansb]

The first option could be useful then.
Additionally, there are many paper on same size justification in Bayesian statistics, you might find something there (I am only familiar with the paper in the second option, but that is limited to Bayes factors as measures of statical evidence).

[DominiqueMakowski]

I think it all boils down to the fact that indeed, in the Bayesian framework, one needs to think differently than in the frequentist one, and there is no straightforward equivalence to "power analysis", as one is able to quantify the evidence in favour, or against, given the available observation.

That said, here are some interesting links:

• https://stats.stackexchange.com/questions/65754/is-power-analysis-necessary-in-bayesian-statistics
• A blogpost about BFDA mentioned above

[cordphelps]

I am so sorry I mentioned 'power'. The point of the question is about how to assess appropriate / effective sample size.

[DominiqueMakowski]

No need to be sorry :)

Do you mean actual sample size (as in number of participants) or actually "effective sample size" (which is a thing on its own in the Bayesian framework)?

[cordphelps]

I'm referring to the number of observations that form the 'evidence'.

(I see I inadvertently chose the adjective 'effective'. I'm not talking about https://en.wikipedia.org/wiki/Effective_sample_size. ugh!)

[cordphelps]

sample size investigation (more info)

bottom line: in a binomial distribution context, I'm feeling more confident about comparing two trials that use similar sample sizes rather than comparing the binomial modes derived from trials that contain dissimilar sample sizes. For a reasonable comparison, I sense (from the figure below) that a "valid sample size" should be > 30.

question: am I making a statistically unsound conclusions?

• I have two trials ('SNH' and 'control')
• each has 270 observations: mostly 0's, some small counts (counts less than 5)
• I want to know if these two trials come from the same source population AND I want to know how many observations are really necessary to get this insight.
• I convert the trial data to binomial data (0s and 1s)
• I base::sample() the 270 observations that I use to define a 31 observation 'prior' for each trial
• for each trial, I create 90 posteriors by chopping the actual trial data into sequentially increasing multiples of 3 observations (posterior 1 has 3 observations, posterior 2 has 6 observations, ....)
• I calculate the mode of each posterior
• I observe that the mode varies with the number of posterior observations and that it seems to want to 'flatten' as I approach 270 observations
• I observe that difference between the modes of the SNH and control trials seems to remain roughly constant

prior and posterior visuals, code and data : https://github.com/cordphelps/hdi