Normalization constants

[sethaxen]

The goal of this issue is to discuss interfaces for dropping expensive/intractible normalization constants.

Each density $p(x|\theta)$ can be broken down into $p(x|\theta) = f(x, \theta) g(\theta) c$:

• $f(x, \theta)$: a "kernel". This contains everything needed for certain methods, generalize those for which $\theta$ is fixed, e.g. $f$ can be used as a prior in MCMC if $\theta$ is fixed, and it can be a proposal/target in self-normalized importance sampling of $x$.
• $g(\theta)$: Generally a scaled integral over $f(x, \theta)$. This is usually intractible, though for a select few named distributions it can be computed (sometimes expensively) using special functions. This is required whenever $\theta$ is not fixed, and for many distributions and inference methods it must be differentiable.
• $c$ is a constant factor dependent on the manifold and choice of measure (and its normalization convention). It's only needed if the method requires that the pdf be normalized to 1. The only method I can think of that would require this is importance sampling (not self-normalized).

In Distributions' own implementations, pdf always computes the full $p$. To reduce the cost of repeated evaluations of pdf, $c g(\theta)$ is often pre-computed in the constructor.

In adhering to Distributions' own API, we can't assume what users will use pdf/logpdf for, so we would ideally return the full $p$. But this is often very expensive or intractible. We could use keywords to pdf/logpdf to specify which is computed, but we can't expect downstream packages to have support for those (e.g. Turing users never call logpdf themselves).

Currently the best way I can think of to support this is either

1. Every family gets a field that specifies its normalization.
2. We by default just return the "kernel" and have a wrapper (e.g. Normalized) for any distribution that guarantees normalization constants are computed (or raises an error if not implemented).

(1) provides a nice terse syntax, while (2) increases code complexity.

[mateuszbaran]

Does Turing.jl currently use non-normalized densities in some way?

[sethaxen]

Since Distributions has no way to specify the normalization assumptions, Turing assumes all densities are normalized. So e.g. if someone implements a custom distribution and its density as just $f(x, \theta)$, and then $\theta$ itself is inferred in a model, the result will be silently wrong. Not a problem when the vast majority of distributions used are defined in Distributions (and are all normalized). But a problem to consider when we're defining distributions.

[sethaxen]

Might be worth giving JuliaStats/Distributions.jl#153 a bump; it's been almost a decade.

[mateuszbaran]

I see, then it looks like this is primarily a Distributions.jl/Turing.jl issue. It looks like we have to return the normalized densities here. Is there a distribution you'd like to add here that has a problematic normalization?

[sethaxen]

It looks like we have to return the normalized densities here.

Not necessarily, so long as we can either decorate the distribution to specify the normalization level and clearly document it. But Distributions handling this would be better.

Is there a distribution you'd like to add here that has a problematic normalization?

Pretty much any distribution on Stiefel except the normalized invariant distribution. e.g. in ManifoldMeasures I had this placeholder where I needed log-pFq for both matrix von Mises-fisher and matrix Bingham distributions: https://github.com/JuliaManifolds/ManifoldMeasures.jl/blob/1c1f6018f14065e7a255c1482ed3b1aaee0fa360/src/specialfunctions.jl#L3-L12