In information theory, joint entropy is a measure of the uncertainty associated with a set of variables.
Definition The joint Shannon entropy (in bits) of two discrete random variables X {\displaystyle X} and Y {\displaystyle Y} with images X {\displaystyle {\mathcal {X}}} and Y {\displaystyle {\mathcal {Y}}} is defined as
H ( X , Y ) = − ∑ x ∈ X ∑ y ∈ Y P ( x , y ) log 2 [ P ( x , y ) ] {\displaystyle \mathrm {H} (X,Y)=-\sum _{x\in {\mathcal {X}}}\sum _{y\in {\mathcal {Y}}}P(x,y)\log _{2}[P(x,y)]}
where x {\displaystyle x} and y {\displaystyle y} are particular values of X {\displaystyle X} and Y {\displaystyle Y} , respectively, P ( x , y ) {\displaystyle P(x,y)} is the joint probability of these values occurring together, and P ( x , y ) log 2 [ P ( x , y ) ] {\displaystyle P(x,y)\log _{2}[P(x,y)]} is defined to be 0 if P ( x , y ) = 0 {\displaystyle P(x,y)=0} . For more than two random variables X 1 , . . . , X n {\displaystyle X_{1},...,X_{n}} this expands to
H ( X 1 , . . . , X n ) = − ∑ x 1 ∈ X 1 . . . ∑ x n ∈ X n P ( x 1 , . . . , x n ) log 2 [ P ( x 1 , . . . , x n ) ] {\displaystyle \mathrm {H} (X_{1},...,X_{n})=-\sum _{x_{1}\in {\mathcal {X}}_{1}}...\sum _{x_{n}\in {\mathcal {X}}_{n}}P(x_{1},...,x_{n})\log _{2}[P(x_{1},...,x_{n})]}
where x 1 , . . . , x n {\displaystyle x_{1},...,x_{n}} are particular values of X 1 , . . . , X n {\displaystyle X_{1},...,X_{n}} , respectively, P ( x 1 , . . . , x n ) {\displaystyle P(x_{1},...,x_{n})} is the probability of these values occurring together, and P ( x 1 , . . . , x n ) log 2 [ P ( x 1 , . . . , x n ) ] {\displaystyle P(x_{1},...,x_{n})\log _{2}[P(x_{1},...,x_{n})]} is defined to be 0 if P ( x 1 , . . . , x n ) = 0 {\displaystyle P(x_{1},...,x_{n})=0} .
Properties
Nonnegativity The joint entropy of a set of random variables is a nonnegative number.
H ( X , Y ) ≥ 0 {\displaystyle \mathrm {H} (X,Y)\geq 0}
… excerpt ends here. Continue reading the full article.
