The information bottleneck method is a technique introduced by Naftali Tishby et al. for finding the best tradeoff between accuracy and complexity (compression) when summarizing (e.g. clustering) a random variable X, given a joint probability distribution between X and an observed relevant variable Y. Other applications include distributional clustering, and dimension reduction. In a well defined sense it generalized the classical notion of minimal sufficient statistics from parametric statistics to arbitrary distributions, not necessarily of exponential form. It does so by relaxing the sufficiency condition to capture some fraction of the mutual information with the relevant variable Y.
The compressed variable is and the algorithm minimises the following quantity
where are the mutual information between and respectively.
Read more about Information Bottleneck Method: Gaussian Information Bottleneck, Defining Decision Contours, Bibliography
Famous quotes containing the words information and/or method:
“Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information upon it.”
—Samuel Johnson (1709–1784)
“I am not afraid of the priests in the long-run. Scientific method is the white ant which will slowly but surely destroy their fortifications. And the importance of scientific method in modern practical life—always growing and increasing—is the guarantee for the gradual emancipation of the ignorant upper and lower classes, the former of whom especially are the strength of the priests.”
—Thomas Henry Huxley (1825–95)