Look into multiple point estimates of moments

[MansMeg]

There is a need after discussion with @bob-carpenter to have point estimates of moments with many more draws that the current 10k reference draws. At the same time, we most likely want to compare many posteriors with similar number of reference draws. Hence, one solution is to store multiple reference draws based on different numbers of reference draws, say 10k, 100k, 1M, 10M etc.

A first step would be to think of how to incorporate this structure intuitively and then add many more point estimates.

[bob-carpenter]

one solution is to store multiple reference draws based on different numbers of reference draws, say 10k, 100k, 1M, 10M etc.

I don't see any reason why we wouldn't always want to use the most reference draws we can extract. Fewer reference draws have larger standard errors and thus introduce more bias into the evaluation. I then think we want to keep to evaluating situations where the bias won't be a big factor. For example, I found bias to be about 5% when evaluating 1000 draws using 10K reference ESS vs. 100K.

I'm also concerned that we not just get independent draws for means of parameters, but also for squares of parameters and for the log density and its square. So I'm running not to a given number of reference draws, but to a minimum ESS target for every expectation of interest, which for me is Theta, Theta^2,log p(Theta), and (log p(Theta))^2. With NUTS, we often have an order of magnitude higher ESS for E[Theta | y] than for E[Theta^2 | y].

[MansMeg]

The number of (approximately independent) draws we can extract will depend on the different models. Some models we might need to run for days to get 10k, while others we get 10k draw in a couple of seconds. Hence, if we just let them run to get the maximum possible draws, we will end up with posterior mean estimates of different quality. My gut feeling says that this is not good and that people would like to have the same number of draws for the set they want to study with the same quality for all posteriors, or?

Hmmm... We now have defined it as the autocorrelation of the draws for parameters to be smaller than 0.05 for all parameters. We can definitely define this broader, and include that we should have low autocorrelation for all four estimates you mention. There should be some theory we could rely on here. Ie maybe we just need to put a constrain on theta^2 and log p(theta|y)^2 if that would bound the autocorrelation yheta and log p(theta|y).

[bob-carpenter]

My gut feeling says that this is not good and that people would like to have the same number of draws

Why? I would prefer the best quality estimates I can get for ground truth. What's to be gained by making some worse than they need to be?

I don't think there are any theoretical guarantees that (log p(theta | y))^2 will be the slowest mixing quantity. Especially not when we can define whatever we want in generated quantities.

An autocorrelation of 0.05 is still really high and should lead to an integrated autocorrelation time (IAT) of at least 1.1, so 10K draws would have an ESS of about 9K. But if this is measuring just the variables and not their squares, the ESS for squared parameters will probably be under 1K.

I can believe that for some model/data combinations we won't be able to generate an ESS greater than 10K for all quantities of interest in a reasonable time and will have to make due with what we can generate. I think that's an even stronger reason not to insist that the ESS is the same for each model/data pair.

[MansMeg]

My gut feeling is that people want comparable quality when running benchmark comparisons between posteriors. If you would have different accuracy on different posteriors, I assume people (or reviewers) will start to ask if difference between posterior results come from different reference moment quality or differences in posterior geometries. Hence, since adding point estimates of moments is not that costly, I think we could include both. Or?

I agree that 0.05 is higherish. But its the cap that all variables should be below in the paper. We could probably change this to improve the quality of the reference draws. I guess below 0.01 in autocorrelation for theta^2 would bump quality quite a bit. If we can get that for most posteriors with reasonable compute, that is.

The question then become where do we draw the line. What is a minimal viable reference draws quality to include as reference draws?

[bob-carpenter]

people (or reviewers) will start to ask if difference between posterior results come from different reference moment quality or differences in posterior geometries.

I can see why you might think reviewers might think that. I try not to worry about reviewers and instead try to get the best answers, but that's not a luxury more junior people have when they need to publish in order to get jobs/promotions. In any case, I don't mind much what you do here as I'm going to run all my own reference values just to be safe and to get what I think is best, even if it does run afoul of reviewer 2's intuitions about fairness. I have plenty of compute. I'm happy to share the results I get, but it sounds like you want something more uniform, and I'm not going to try to control everything to have exactly the same standardized error.

I think it'd help to report standardized standard errors on any reference estimates, but even that's contentious and maybe just standard errors that don't rely on an additional estimate of variance? We just want to make sure that the standard error in our reference doesn't dominate the evaluations.

What is a minimal viable reference draws quality

I'm aiming at an ESS of 10K to 100K for all of E[X], E[X^2], E[X^4], E[log p(X)], E[(log p(X))^2] and E[(log p(X))^4]. I have a lot of compute at my disposal here and don't need to use every model in posteriordb. I still may not be able to get to 10K on all models.

I don't know how large and independent the reference has to be to remove bias in things like the Wasserstein distance calculation we were using for Pathfinder evals, for example. This 2025 AISTATS paper by Agrawal and Domke compares i.i.d. draws to the reference to get a sense of the reference quality for Wasserstein distances.

[MansMeg]

I fully agree with that we should not care too much about reviewer 2 and focus on good research. I’m just not sure how to weigh comparability vs accuracy in the best possible way. You might be right here and accuracy should by our main concern, I honestly don't know. We should definitely discuss this next time we meet. Im happy to be convinced either way.

So my thoughts is more if we can have the best of two worlds, ie both the best accuracy point estimates and point estimates with comparable quality. And how would we then inplement that in the best possible way?

Great! Ill read that paper.

[bob-carpenter]

If you give users a choice of maximum quality vs. equal quality, then they have to understand how to make that choice and now have an extra note in their evaluation section behind "used posteriordb". It's not the end of the world, but it's not free.

[MansMeg]

Yes. We should definitely discuss this when we meet next week. Im really unsure how to think here. Hence a discussion with the group is probably good, to see what others think.