The information bottleneck method is a technique introduced by Naftali Tishby et al. for finding the best tradeoff between accuracy and complexity (compression) when summarizing (e.g. clustering) a random variable X, given a joint probability distribution between X and an observed relevant variable Y. Other applications include distributional clustering, and dimension reduction. In a well defined sense it generalized the classical notion of minimal sufficient statistics from parametric statistics to arbitrary distributions, not necessarily of exponential form. It does so by relaxing the sufficiency condition to capture some fraction of the mutual information with the relevant variable Y.
The compressed variable is and the algorithm minimises the following quantity
where are the mutual information between and respectively.
Read more about Information Bottleneck Method: Gaussian Information Bottleneck, Defining Decision Contours, Bibliography
Famous quotes containing the words information and/or method:
“I was brought up to believe that the only thing worth doing was to add to the sum of accurate information in the world.”
—Margaret Mead (1901–1978)
“... [a] girl one day flared out and told the principal “the only mission opening before a girl in his school was to marry one of those candidates [for the ministry].” He said he didn’t know but it was. And when at last that same girl announced her desire and intention to go to college it was received with about the same incredulity and dismay as if a brass button on one of those candidate’s coats had propounded a new method for squaring the circle or trisecting the arc.”
—Anna Julia Cooper (1859–1964)