Hard Integrals

The “evidence” or “marginal likelihood” integral, $p(x) = \int p(\theta)p(x | \theta) \, d\theta$, is often considered hard to calculate. The most general method I know of is Nested Sampling, although sometimes other methods can outperform it if the shape of the likelihood function is kind to the method.

However, there are other integrals that Bayesians need to do that are orders of magnitude harder than this. Suppose I want to measure some parameters $\theta$ from some data $x$, and there are also nuisance parameters $\eta$ in the problem. The amount of information learned, $\mathcal{H}$, is quantified by how compressed the posterior is relative to the prior:

$\mathcal{H} = \int p(\theta | x) \log\left[\frac{p(\theta | x)}{p(\theta)}\right] \, d\theta$

Remembering that we have nuisance parameters, the posterior and prior in this expression may be written as marginalisations of the joint posterior/prior of $\theta$ and $\eta$.

$\mathcal{H} = \int \left(\int p(\eta, \theta | x)\, d\eta\right) \log\left[\frac{\left(\int p(\eta, \theta | x) \, d\eta\right)}{\left(\int p(\eta, \theta)\,d\eta \right)}\right] \, d\theta$

That’s how much information the data provided to us about the parameters of interest. Before we get the data, we might be interested in knowing the expected amount of information we will get, $\left< \mathcal{H} \right>$, with the expectation being taken with respect to the prior for the data, $p(x) = \int p(\theta)p(x|\theta) \, d\theta$:

$\left<\mathcal{H}\right>= \int p(x) \int \left(\int p(\eta, \theta | x)\, d\eta\right) \log\left[\frac{\left(\int p(\eta, \theta | x) \, d\eta\right)}{\left(\int p(\eta, \theta)\,d\eta \right)}\right] \, d\theta \, dx$

We can use this to do experimental design: it makes sense to choose your experiment to optimise the expected amount of information you’ll get about the parameters of interest. This wraps an optimisation problem around the above expression too.

In summary, the evidence is not a hard integral. We know how to do it. Other useful integrals, not so much.