Entropy (information Theory) - Further Properties

Further Properties

The Shannon entropy satisfies the following properties, for some of which it is useful to interpret entropy as the amount of information learned (or uncertainty eliminated) by revealing the value of a random variable X:

• Adding or removing an event with probability zero does not contribute to the entropy:

.

• It can be confirmed using the Jensen inequality that

.

This maximal entropy of is effectively attained by a source alphabet having a uniform probability distribution: uncertainty is maximal when all possible events are equiprobable.

• The entropy or the amount of information revealed by evaluating (X,Y) (that is, evaluating X and Y simultaneously) is equal to the information revealed by conducting two consecutive experiments: first evaluating the value of Y, then revealing the value of X given that you know the value of Y. This may be written as

• If Y=f(X) where f is deterministic, then . Applying the previous formula to yields

, so ,

thus the entropy of a variable can only decrease when the latter is passed through a deterministic function.

• If X and Y are two independent experiments, then knowing the value of Y doesn't influence our knowledge of the value of X (since the two don't influence each other by independence):

• The entropy of two simultaneous events is no more than the sum of the entropies of each individual event, and are equal if the two events are independent. More specifically, if X and Y are two random variables on the same probability space, and (X,Y) denotes their Cartesian product, then

Proving this mathematically follows easily from the previous two properties of entropy.