Hard Integrals

The “evidence” or “marginal likelihood” integral, p(x) = \int p(\theta)p(x | \theta) \, d\theta, is often considered hard to calculate. The most general method I know of is Nested Sampling, although sometimes other methods can outperform it if the shape of the likelihood function is kind to the method.

However, there are other integrals that Bayesians need to do that are orders of magnitude harder than this. Suppose I want to measure some parameters \theta from some data x, and there are also nuisance parameters \eta in the problem. The amount of information learned, \mathcal{H}, is quantified by how compressed the posterior is relative to the prior:

\mathcal{H} = \int p(\theta | x) \log\left[\frac{p(\theta | x)}{p(\theta)}\right] \, d\theta

Remembering that we have nuisance parameters, the posterior and prior in this expression may be written as marginalisations of the joint posterior/prior of \theta and \eta.

\mathcal{H} = \int \left(\int p(\eta, \theta | x)\, d\eta\right) \log\left[\frac{\left(\int p(\eta, \theta | x) \, d\eta\right)}{\left(\int p(\eta, \theta)\,d\eta \right)}\right] \, d\theta

That’s how much information the data provided to us about the parameters of interest. Before we get the data, we might be interested in knowing the expected amount of information we will get, \left< \mathcal{H} \right>, with the expectation being taken with respect to the prior for the data, p(x) = \int p(\theta)p(x|\theta) \, d\theta:

\left<\mathcal{H}\right>= \int p(x) \int \left(\int p(\eta, \theta | x)\, d\eta\right) \log\left[\frac{\left(\int p(\eta, \theta | x) \, d\eta\right)}{\left(\int p(\eta, \theta)\,d\eta \right)}\right] \, d\theta \, dx

We can use this to do experimental design: it makes sense to choose your experiment to optimise the expected amount of information you’ll get about the parameters of interest. This wraps an optimisation problem around the above expression too.

In summary, the evidence is not a hard integral. We know how to do it. Other useful integrals, not so much.

Advertisements

About Brendon J. Brewer

I am a senior lecturer in the Department of Statistics at The University of Auckland. Any opinions expressed here are mine and are not endorsed by my employer.
This entry was posted in Entropy, Inference, Information and tagged . Bookmark the permalink.

One Response to Hard Integrals

  1. Pingback: Brendon Brewer on “Hard Integrals” | Hypergeometric

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s