In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as shannons (bits), nats or hartleys) obtained about one random variable by observing the other random variable. The concept of mutual information is intimately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable. Not limited to real-valued random variables and linear dependence like the correlation coefficient, MI is more general and determines how different the joint distribution of the pair ( X , Y ) {\displaystyle (X,Y)} is from the product of the marginal distributions of X {\displaystyle X} and Y {\displaystyle Y} . MI is the expected value of the pointwise mutual information (PMI). The quantity was defined and analyzed by Claude Shannon in his landmark paper "A Mathematical Theory of Communication", although he did not call it "mutual information". This term was coined later by Robert Fano. Mutual Information is also known as information gain.
Definition Let ( X , Y ) {\displaystyle (X,Y)} be a pair of random variables with values over the space X × Y {\displaystyle {\mathcal {X}}\times {\mathcal {Y}}} . If their joint distribution is P ( X , Y ) {\displaystyle P_{(X,Y)}} and the marginal distributions are P X {\displaystyle P_{X}} and P Y {\displaystyle P_{Y}} , the mutual information is defined as
I ( X ; Y ) = D K L ( P ( X , Y ) ∥ P X ⊗ P Y ) {\displaystyle I(X;Y)=D_{\mathrm {KL} }(P_{(X,Y)}\parallel P_{X}\otimes P_{Y})}
where D K L {\displaystyle D_{\mathrm {KL} }} is the Kullback–Leibler divergence, and P X ⊗ P Y {\displaystyle P_{X}\otimes P_{Y}} is the outer product distribution which assigns probability P X ( x ) ⋅ P Y ( y ) {\displaystyle P_{X}(x)\cdot P_{Y}(y)} to each ( x , y ) {\displaystyle (x,y)} . Expressed in terms of the entropy H ( ⋅ ) {\displaystyle H(\cdot )} and the conditional entropy H ( ⋅ | ⋅ ) {\displaystyle H(\cdot |\cdot )} of the random variables X {\displaystyle X} and Y {\displaystyle Y} , one also has (see relation to conditional and joint entropy):
I ( X ; Y ) = H ( X ) − H ( X | Y ) = H ( Y ) − H ( Y | X ) {\displaystyle I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)}
Notice, as per property of the Kullback–Leibler divergence, that I ( X ; Y ) {\displaystyle I(X;Y)} is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when X {\displaystyle X} and Y {\displaystyle Y} are independent (and hence observing Y {\displaystyle Y} tells you nothing about X {\displaystyle X} ). I ( X ; Y ) {\displaystyle I(X;Y)} is non-negative. It is a measure of the price for encoding ( X , Y ) {\displaystyle (X,Y)} as a pair of independent random variables when in reality they are not. If the natural logarithm is used, the unit of mutual information is the nat. If the log base 2 is used, the unit of mutual information is the shannon, also known as the bit. If the log base 10 is used, the unit of mutual information is the hartley, also known as the ban or the dit.
In terms of PMFs for discrete distributions The mutual information of two jointly discrete random variables X {\displaystyle X} and Y {\displaystyle Y} is calculated as a double sum:
I ( X ; Y ) = ∑ y ∈ Y ∑ x ∈ X P ( X , Y ) ( x , y ) log ( P ( X , Y ) ( x , y ) P X ( x ) P Y ( y ) ) {\displaystyle \operatorname {I} (X;Y)=\sum _{y\in {\mathcal {Y}}}\sum _{x\in {\mathcal {X}}}{P_{(X,Y)}(x,y)\log \left({\frac {P_{(X,Y)}(x,y)}{P_{X}(x)\,P_{Y}(y)}}\right)}} , where P ( X , Y ) {\displaystyle P_{(X,Y)}} is the joint probability mass function of X {\displaystyle X} and Y {\displaystyle Y} , and P X {\displaystyle P_{X}} and P Y {\displaystyle P_{Y}} are the marginal probability mass functions of X {\displaystyle X} and Y {\displaystyle Y} respectively.
In terms of PDFs for continuous distributions In the case of jointly continuous random variables, the double sum is replaced by a double integral:
I ( X ; Y ) = ∫ Y ∫ X P ( X , Y ) ( x , y ) log ( P ( X , Y ) ( x , y ) P X ( x ) P Y ( y ) ) d x d y {\displaystyle \operatorname {I} (X;Y)=\int _{\mathcal {Y}}\int _{\mathcal {X}}{P_{(X,Y)}(x,y)\log {\left({\frac {P_{(X,Y)}(x,y)}{P_{X}(x)\,P_{Y}(y)}}\right)}}\;dx\,dy} , where P ( X , Y ) {\displaystyle P_{(X,Y)}} is now the joint probability density function of X {\displaystyle X} and Y {\displaystyle Y} , and P X {\displaystyle P_{X}} and P Y {\displaystyle P_{Y}} are the marginal probability density functions of X {\displaystyle X} and Y {\displaystyle Y} respectively.
Motivation Intuitively, mutual information measures the information that X {\displaystyle X} and Y {\displaystyle Y} share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if X {\displaystyle X} and Y {\displaystyle Y} are independent, then knowing X {\displaystyle X} does not give any information about Y {\displaystyle Y} and vice versa, so their mutual information is zero. At the other extreme, if X {\displaystyle X} is a deterministic function of Y {\displaystyle Y} and Y {\displaystyle Y} is a deterministic function of X {\displaystyle X} then all information conveyed by X {\displaystyle X} is shared with Y {\displaystyle Y} : knowing X {\displaystyle X} determines the value of Y {\displaystyle Y} and vice versa. As a result, the mutual information is the same as the uncertainty contained in Y {\displaystyle Y} (or X {\displaystyle X} ) alone, namely the entropy of Y {\displaystyle Y} (or X {\displaystyle X} ). A very special case of this is when X {\displaystyle X} and Y {\displaystyle Y} are the same random variable. Mutual information is a measure of the inherent dependence expressed in the joint distribution of X {\displaystyle X} and Y {\displaystyle Y} relative to the marginal distribution of X {\displaystyle X} and Y {\displaystyle Y} under the assumption of independence. Mutual information therefore measures dependence in the following sense: I ( X ; Y ) = 0 {\displaystyle \operatorname {I} (X;Y)=0} if and only if X {\displaystyle X} and Y {\displaystyle Y} are independent random variables. This is easy to see in one direction: if X {\displaystyle X} and Y {\displaystyle Y} are independent, then p ( X , Y ) ( x , y ) = p X ( x ) ⋅ p Y ( y ) {\displaystyle p_{(X,Y)}(x,y)=p_{X}(x)\cdot p_{Y}(y)} , and therefore:
log ( p ( X , Y ) ( x , y ) p X ( x ) p Y ( y ) ) = log 1 = 0 {\displaystyle \log {\left({\frac {p_{(X,Y)}(x,y)}{p_{X}(x)\,p_{Y}(y)}}\right)}=\log 1=0} . Moreover, mutual information is nonnegative (i.e. I ( X ; Y ) ≥ 0 {\displaystyle \operatorname {I} (X;Y)\geq 0} see below) and symmetric (i.e. I ( X ; Y ) = I ( Y ; X ) {\displaystyle \operatorname {I} (X;Y)=\operatorname {I} (Y;X)} see below).
Properties
Nonnegativity Using Jensen's inequality on the definition of mutual information we can show that I ( X ; Y ) {\displaystyle \operatorname {I} (X;Y)} is non-negative, i.e.
I ( X ; Y ) ≥ 0 {\displaystyle \operatorname {I} (X;Y)\geq 0}
Symmetry
I ( X ; Y ) = I ( Y ; X ) {\displaystyle \operatorname {I} (X;Y)=\operatorname {I} (Y;X)}
The proof is given considering the relationship with entropy, as shown below.
Supermodularity under independence If C {\displaystyle C} is independent of ( A , B ) {\displaystyle (A,B)} , then
I ( Y ; A , B , C ) − I ( Y ; A , B ) ≥ I ( Y ; A , C ) − I ( Y ; A ) {\displaystyle \operatorname {I} (Y;A,B,C)-\operatorname {I} (Y;A,B)\geq \operatorname {I} (Y;A,C)-\operatorname {I} (Y;A)} .
Relation to conditional and joint entropy Mutual information can be equivalently expressed as:
I ( X ; Y )
≡ H ( X ) − H ( X ∣ Y )
≡ H ( Y ) − H ( Y ∣ X )
≡ H ( X ) + H ( Y ) − H ( X , Y )
≡ H ( X , Y ) − H ( X ∣ Y ) − H ( Y ∣ X ) {\displaystyle {\begin{aligned}\operatorname {I} (X;Y)&{}\equiv \mathrm {H} (X)-\mathrm {H} (X\mid Y)\\&{}\equiv \mathrm {H} (Y)-\mathrm {H} (Y\mid X)\\&{}\equiv \mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y)\\&{}\equiv \mathrm {H} (X,Y)-\mathrm {H} (X\mid Y)-\mathrm {H} (Y\mid X)\end{aligned}}}
where H ( X ) {\displaystyle \mathrm {H} (X)} and H ( Y ) {\displaystyle \mathrm {H} (Y)} are the marginal entropies, H ( X ∣ Y ) {\displaystyle \mathrm {H} (X\mid Y)} and H ( Y ∣ X ) {\displaystyle \mathrm {H} (Y\mid X)} are the conditional entropies, and H ( X , Y ) {\displaystyle \mathrm {H} (X,Y)} is the joint entropy of X {\displaystyle X} and Y {\displaystyle Y} . Notice the analogy to the union, difference, and intersection of two sets: in this respect, all the formulas given above are apparent from the Venn diagram reported at the beginning of the article. In terms of a communication channel in which the output Y {\displaystyle Y} is a noisy version of the input X {\displaystyle X} , these relations are summarised in the figure:
Because I ( X ; Y ) {\displaystyle \operatorname {I} (X;Y)} is non-negative, consequently, H ( X ) ≥ H ( X ∣ Y ) {\displaystyle \mathrm {H} (X)\geq \mathrm {H} (X\mid Y)} . Here we give the detailed deduction of I ( X ; Y ) = H ( Y ) − H ( Y ∣ X ) {\displaystyle \operatorname {I} (X;Y)=\mathrm {H} (Y)-\mathrm {H} (Y\mid X)} for the case of jointly discrete random variables:
I ( X ; Y )
= ∑ x ∈ X , y ∈ Y p ( X , Y ) ( x , y ) log p ( X , Y ) ( x , y ) p X ( x ) p Y ( y )
= ∑ x ∈ X , y ∈ Y p ( X , Y ) ( x , y ) log p ( X , Y ) ( x , y ) p X ( x ) − ∑ x ∈ X , y ∈ Y p ( X , Y ) ( x , y ) log p Y ( y )
= ∑ x ∈ X , y ∈ Y p X ( x ) p Y ∣ X = x ( y ) log p Y ∣ X = x ( y ) − ∑ x ∈ X , y ∈ Y p ( X , Y ) ( x , y ) log p Y ( y )
= ∑ x ∈ X p X ( x ) ( ∑ y ∈ Y p Y ∣ X = x ( y ) log p Y ∣ X = x ( y ) ) − ∑ y ∈ Y ( ∑ x ∈ X p ( X , Y ) ( x , y ) ) log p Y ( y )
= − ∑ x ∈ X p X ( x ) H ( Y ∣ X = x ) − ∑ y ∈ Y p Y ( y ) log p Y ( y )
= − H ( Y ∣ X ) + H ( Y )
= H ( Y ) − H ( Y ∣ X ) . {\displaystyle {\begin{aligned}\operatorname {I} (X;Y)&{}=\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p_{(X,Y)}(x,y)\log {\frac {p_{(X,Y)}(x,y)}{p_{X}(x)p_{Y}(y)}}\\&{}=\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p_{(X,Y)}(x,y)\log {\frac {p_{(X,Y)}(x,y)}{p_{X}(x)}}-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p_{(X,Y)}(x,y)\log p_{Y}(y)\\&{}=\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p_{X}(x)p_{Y\mid X=x}(y)\log p_{Y\mid X=x}(y)-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p_{(X,Y)}(x,y)\log p_{Y}(y)\\&{}=\sum _{x\in {\mathcal {X}}}p_{X}(x)\left(\sum _{y\in {\mathcal {Y}}}p_{Y\mid X=x}(y)\log p_{Y\mid X=x}(y)\right)-\sum _{y\in {\mathcal {Y}}}\left(\sum _{x\in {\mathcal {X}}}p_{(X,Y)}(x,y)\right)\log p_{Y}(y)\\&{}=-\sum _{x\in {\mathcal {X}}}p_{X}(x)\mathrm {H} (Y\mid X=x)-\sum _{y\in {\mathcal {Y}}}p_{Y}(y)\log p_{Y}(y)\\&{}=-\mathrm {H} (Y\mid X)+\mathrm {H} (Y)\\&{}=\mathrm {H} (Y)-\mathrm {H} (Y\mid X).\\\end{aligned}}}
The proofs of the other identities above are similar. The proof of the general case (not just discrete) is similar, with integrals replacing sums. Intuitively, if entropy H ( Y ) {\displaystyle \mathrm {H} (Y)} is regarded as a measure of uncertainty about a random variable, then H ( Y ∣ X ) {\displaystyle \mathrm {H} (Y\mid X)} is a measure of what X {\displaystyle X} does not say about Y {\displaystyle Y} . This is "the amount of uncertainty remaining about Y {\displaystyle Y} after X {\displaystyle X} is known", and thus the right side of the second of these equalities can be read as "the amount of uncertainty in Y {\displaystyle Y} , minus the amount of uncertainty in Y {\displaystyle Y} which remains after X {\displaystyle X} is known", which is equivalent to "the amount of uncertainty in Y {\displaystyle Y} which is removed by knowing X {\displaystyle X} ". This corroborates the intuitive meaning of mutual information as the amount of information (that is, reduction in uncertainty) that knowing either variable provides about the other. Note that in the discrete case H ( Y ∣ Y ) = 0 {\displaystyle \mathrm {H} (Y\mid Y)=0} and therefore H ( Y ) = I ( Y ; Y ) {\displaystyle \mathrm {H} (Y)=\operatorname {I} (Y;Y)} . Thus I ( Y ; Y ) ≥ I ( X ; Y ) {\displaystyle \operatorname {I} (Y;Y)\geq \operatorname {I} (X;Y)} , and one can formulate the basic principle that a variable contains at least as much information about itself as any other variable can provide.
Relation to Kullback–Leibler divergence For jointly discrete or jointly continuous pairs ( X , Y ) {\displaystyle (X,Y)} , mutual information is the Kullback–Leibler divergence from the product of the marginal distributions, p X ⋅ p Y {\displaystyle p_{X}\cdot p_{Y}} , of the joint distribution p ( X , Y ) {\displaystyle p_{(X,Y)}} , that is,
I ( X ; Y ) = D KL ( p ( X , Y ) ∥ p X p Y )