In information theory, the cross entropy between two probability distributions measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution, rather than the "true" distribution .
The cross entropy for two distributions and over the same probability space is thus defined as follows:
- ,
where is the entropy of, and is the Kullback-Leibler divergence of from (also known as the relative entropy).
For discrete and this means
The situation for continuous distributions is analogous:
NB: The notation is sometimes used for both the cross entropy as well as the joint entropy of and .
Read more about Cross Entropy: Motivation, Estimation, Cross-entropy Minimization
Famous quotes containing the words cross and/or entropy:
“I know we’re not saints or virgins or lunatics; we know all the lust and lavatory jokes, and most of the dirty people; we can catch buses and count our change and cross the roads and talk real sentences. But our innocence goes awfully deep, and our discreditable secret is that we don’t know anything at all, and our horrid inner secret is that we don’t care that we don’t.”
—Dylan Thomas (1914–1953)
“Just as the constant increase of entropy is the basic law of the universe, so it is the basic law of life to be ever more highly structured and to struggle against entropy.”
—Václav Havel (b. 1936)