Conjugate Prior - Example

Example

The form of the conjugate prior can generally be determined by inspection of the probability density or probability mass function of a distribution. For example, consider a random variable which is a Bernoulli trial with unknown probability of success q in . The probability density function has the form

Expressed as a function of, this has the form

for some constants and . Generally, this functional form will have an additional multiplicative factor (the normalizing constant) ensuring that the function is a probability distribution, i.e. the integral over the entire range is 1. This factor will often be a function of and, but never of .

In fact, the usual conjugate prior is the beta distribution with

where and are chosen to reflect any existing belief or information ( = 1 and = 1 would give a uniform distribution) and Β(, ) is the Beta function acting as a normalising constant.

In this context, and are called hyperparameters (parameters of the prior), to distinguish them from parameters of the underlying model (here q). It is a typical characteristic of conjugate priors that the dimensionality of the hyperparameters is one greater than that of the parameters of the original distribution. If all parameters are scalar values, then this means that there will be one more hyperparameter than parameter; but this also applies to vector-valued and matrix-valued parameters. (See the general article on the exponential family, and consider also the Wishart distribution, conjugate prior of the covariance matrix of a multivariate normal distribution, for an example where a large dimensionality is involved.)

If we then sample this random variable and get s successes and f failures, we have

which is another Beta distribution with a simple change to the (hyper)parameters. This posterior distribution could then be used as the prior for more samples, with the hyperparameters simply adding each extra piece of information as it comes.