Differential Entropy - Maximization in The Normal Distribution

Maximization in The Normal Distribution

With a normal distribution, differential entropy is maximized for a given variance. The following is a proof that a Gaussian variable has the largest entropy amongst all random variables of equal variance.

Let g(x) be a Gaussian PDF with mean μ and variance σ2 and f(x) an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that f(x) has the same mean of μ as g(x).

Consider the Kullback-Leibler divergence between the two distributions

Now note that

\begin{align} \int_{-\infty}^\infty f(x)\log(g(x)) dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) dx \\ &= \int_{-\infty}^\infty f(x) \log\frac{1}{\sqrt{2\pi\sigma^2}} dx + \log(e)\int_{-\infty}^\infty f(x)\left( -\frac{(x-\mu)^2}{2\sigma^2}\right) dx \\ &= -\tfrac{1}{2}\log(2\pi\sigma^2) - \log(e)\frac{\sigma^2}{2\sigma^2} \\ &= -\tfrac{1}{2}\left(\log(2\pi\sigma^2) + \log(e)\right) \\ &= -\tfrac{1}{2}\log(2\pi e \sigma^2) \\ &= -h(g) \end{align}

because the result does not depend on f(x) other than through the variance. Combining the two results yields

with equality when g(x) = f(x) following from the properties of Kullback-Leibler divergence.

This result may also be demonstrated using the variational calculus. A Lagrangian function with two Lagrangian multipliers may be defined as:

where g(x) is some function with mean μ. When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition and the requirement of fixed variance, are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero:

Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields:

Using the constraint equations to solve for λ0 and λ yields the normal distribution:

Read more about this topic:  Differential Entropy

Famous quotes containing the words normal and/or distribution:

    “Insecurity, commonly regarded as a weakness in normal people, is the basic tool of the actor’s trade.”
    —Miranda Richardson (b. 1958)

    “Classical and romantic: private language of a family quarrel, a dead dispute over the distribution of emphasis between man and nature.”
    —Cyril Connolly (1903–1974)