Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding reloo #12

Open
sethaxen opened this issue Aug 14, 2023 · 0 comments
Open

Adding reloo #12

sethaxen opened this issue Aug 14, 2023 · 0 comments

Comments

@sethaxen
Copy link
Member

arviz.reloo allows for drawing samples from a LOO posterior for all data points for which the PSIS-LOO pareto shape diagnostic indicates that PSIS-LOO gives poor estimates.

We should think carefully about how to support this. In particular, we need to design an API that can be easily extended by e.g. a PPL but is straightforward to use without a PPL. The result should be an AbstractELPDResult, but we need some way to indicate that this elpd result comes from different estimation methods. Hence the LOO posteriors refitted should probably have a missing pareto shape diagnostic.

But more generally, once we have this functionality, running exact LOO, LOGO, or k-fold CV should be possible, so the API should support that as well.

In addition, the functionality in #8 should be supported, but this is more complicated, because posterior predictions may be required, and because the order of operations is reversed. With PSIS-LOO, we compute approximate LOO once and reuse the importance weights for all expectations from LOO posteriors, while with exact LOO, one loops over each data point held out and computes each expectation for each LOO posterior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant