The information bottleneck method is a technique in information theory introduced by Naftali Tishby, Fernando C. Pereira, and William Bialek. It is designed for finding the best tradeoff between accuracy and complexity (compression) when summarizing (e.g. clustering) a random variable X, given a joint probability distribution p(X,Y) between X and an observed relevant variable Y - and self-described as providing "a surprisingly rich framework for discussing a variety of problems in signal processing and learning". Applications include distributional clustering and dimension reduction, and more recently it has been suggested as a theoretical foundation for deep learning. It generalized the classical notion of minimal sufficient statistics from parametric statistics to arbitrary distributions, not necessarily of exponential form. It does so by relaxing the sufficiency condition to capture some fraction of the mutual information with the relevant variable Y. The information bottleneck can also be viewed as a rate distortion problem, with a distortion function that measures how well Y is predicted from a compressed representation T compared to its direct prediction from X. This interpretation provides a general iterative algorithm for solving the information bottleneck trade-off and calculating the information curve from the distribution p(X,Y). Let the compressed representation be given by random variable T {\displaystyle T} . The algorithm minimizes the following functional with respect to conditional distribution p ( t | x ) {\displaystyle p(t|x)} :
inf p ( t | x ) ( I ( X ; T ) − β I ( T ; Y ) ) , {\displaystyle \inf _{p(t|x)}\,\,{\Big (}I(X;T)-\beta I(T;Y){\Big )},}
where I ( X ; T ) {\displaystyle I(X;T)} and I ( T ; Y ) {\displaystyle I(T;Y)} are the mutual information of X {\displaystyle X} and T {\displaystyle T} , and of T {\displaystyle T} and Y {\displaystyle Y} , respectively, and β {\displaystyle \beta } is a Lagrange multiplier.
Learning theory for deep learning It has been mathematically proven that controlling information bottleneck is one way to control generalization error in deep learning. Namely, the generalization error is proven to scale as O ~ ( I ( X , T ) + 1 n ) {\displaystyle {\tilde {O}}\left({\sqrt {\frac {I(X,T)+1}{n}}}\right)} where n {\displaystyle n} is the number of training samples, X {\displaystyle X} is the input to a deep neural network, and T {\displaystyle T} is the output of a hidden layer. This generalization bound scale with the degree of information bottleneck, unlike the other generalization bounds that scale with the number of parameters, VC dimension, Rademacher complexity, stability or robustness.
Phase transitions
Information theory of deep learning Theory of Information Bottleneck is recently used to study Deep Neural Networks (DNN). Consider X {\displaystyle X} and Y {\displaystyle Y} respectively as the input and output layers of a DNN, and let T {\displaystyle T} be any hidden layer of the network. Shwartz-Ziv and Tishby proposed the information bottleneck that expresses the tradeoff between the mutual information measures I ( X , T ) {\displaystyle I(X,T)} and I ( T , Y ) {\displaystyle I(T,Y)} . In this case, I ( X , T ) {\displaystyle I(X,T)} and I ( T , Y ) {\displaystyle I(T,Y)} respectively quantify the amount of information that the hidden layer contains about the input and the output. They conjectured that the training process of a DNN consists of two separate phases; 1) an initial fitting phase in which I ( T , Y ) {\displaystyle I(T,Y)} increases, and 2) a subsequent compression phase in which I ( X , T ) {\displaystyle I(X,T)} decreases. Saxe et al. in countered the claim of Shwartz-Ziv and Tishby, stating that this compression phenomenon in DNNs is not comprehensive, and it depends on the particular activation function. In particular, they claimed that the compression does not happen with ReLu activation functions. Shwartz-Ziv and Tishby disputed these claims, arguing that Saxe et al. had not observed compression due to weak estimates of the mutual information. On the other hand, recently Goldfeld et al. have argued that the observed compression is a result of geometric, and not of information-theoretic phenomena, a view that has been shared also in.
Variational bottleneck
… excerpt ends here. Continue reading the full article.
