Differential Entropy - Properties of Differential Entropy

Properties of Differential Entropy

• For two densities f and g, the Kullback-Leibler divergence D(f||g) is nonnegative with equality if f = g almost everywhere. Similarly, for two random variables X and Y, I(X;Y) ≥ 0 and h(X|Y) ≤ h(X) with equality if and only if X and Y are independent.
• The chain rule for differential entropy holds as in the discrete case

.

• Differential entropy is translation invariant, i.e., h(X + c) = h(X) for a constant c.
• Differential entropy is in general not invariant under arbitrary invertible maps. In particular, for a constant a, h(aX) = h(X) + log|a|. For a vector valued random variable X and a matrix A, h(A X) = h(X) + log|det(A)|.
• In general, for a transformation from a random vector to another random vector with same dimension Y = m(X), the corresponding entropies are related via

where is the Jacobian of the transformation m. Equality is achieved if the transform is bijective, i.e., invertible.

• If a random vector X in Rn has mean zero and covariance matrix K, with equality if and only if X is jointly gaussian (see below).

However, differential entropy does not have other desirable properties:

• It is not invariant under change of variables.
• It can be negative.

A modification of differential entropy that addresses this is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).