Differential Entropy - Maximization in The Normal Distribution

Maximization in The Normal Distribution

With a normal distribution, differential entropy is maximized for a given variance. The following is a proof that a Gaussian variable has the largest entropy amongst all random variables of equal variance.

Let g(x) be a Gaussian PDF with mean μ and variance σ2 and f(x) an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that f(x) has the same mean of μ as g(x).

Consider the Kullback-Leibler divergence between the two distributions

Now note that

\begin{align} \int_{-\infty}^\infty f(x)\log(g(x)) dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) dx \\ &= \int_{-\infty}^\infty f(x) \log\frac{1}{\sqrt{2\pi\sigma^2}} dx + \log(e)\int_{-\infty}^\infty f(x)\left( -\frac{(x-\mu)^2}{2\sigma^2}\right) dx \\ &= -\tfrac{1}{2}\log(2\pi\sigma^2) - \log(e)\frac{\sigma^2}{2\sigma^2} \\ &= -\tfrac{1}{2}\left(\log(2\pi\sigma^2) + \log(e)\right) \\ &= -\tfrac{1}{2}\log(2\pi e \sigma^2) \\ &= -h(g) \end{align}

because the result does not depend on f(x) other than through the variance. Combining the two results yields

with equality when g(x) = f(x) following from the properties of Kullback-Leibler divergence.

This result may also be demonstrated using the variational calculus. A Lagrangian function with two Lagrangian multipliers may be defined as:

where g(x) is some function with mean μ. When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition and the requirement of fixed variance, are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero:

Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields:

Using the constraint equations to solve for λ0 and λ yields the normal distribution:

Read more about this topic:  Differential Entropy

Famous quotes containing the words normal and/or distribution:

    “Like sleep disturbances, some worries at separation can be expected in the second year. If you accept this, then you will avoid reacting to this anxiety as if it’s your fault. A mother who feels guilty will appear anxious to the child, as if to affirm the child’s anxiety. By contrast, a parent who understands that separation anxiety is normal is more likely to react in a way that soothes and reassures the child.”
    —Cathy Rindner Tempelsman (20th century)

    “The question for the country now is how to secure a more equal distribution of property among the people. There can be no republican institutions with vast masses of property permanently in a few hands, and large masses of voters without property.... Let no man get by inheritance, or by will, more than will produce at four per cent interest an income ... of fifteen thousand dollars] per year, or an estate of five hundred thousand dollars.”
    —Rutherford Birchard Hayes (1822–1893)