In statistics and information theory, a maximum entropy probability distribution has entropy that is at least as great as that of all other members of a specified class of probability distributions. According to the principle of maximum entropy, if nothing is known about a distribution except that it belongs to a certain class (usually defined in terms of specified properties or measures), then the distribution with the largest entropy should be chosen as the least-informative default. The motivation is twofold: first, maximizing entropy minimizes the amount of prior information built into the distribution; second, many physical systems tend to move towards maximal entropy configurations over time.
Definition of entropy and differential entropy
If X {\displaystyle X} is a continuous random variable with probability density p ( x ) {\displaystyle p(x)} , then the differential entropy of X {\displaystyle X} is defined as
H ( X ) = − ∫ − ∞ ∞ p ( x ) log p ( x ) d x . {\displaystyle H(X)=-\int _{-\infty }^{\infty }p(x)\log p(x)\,dx~.}
If X {\displaystyle X} is a discrete random variable with distribution given by
Pr ( X = x k ) = p k for k = 1 , 2 , … {\displaystyle \Pr(X{=}x_{k})=p_{k}\qquad {\text{ for }}\quad k=1,2,\ldots }
then the entropy of X {\displaystyle X} is defined as
H ( X ) = − ∑ k ≥ 1 p k log p k . {\displaystyle H(X)=-\sum _{k\geq 1}p_{k}\log p_{k}\,.}
The seemingly divergent term p ( x ) log p ( x ) {\displaystyle p(x)\log p(x)} is replaced by zero, whenever p ( x ) = 0 . {\displaystyle p(x)=0\,.}
This is a special case of more general forms described in the articles Entropy (information theory), Principle of maximum entropy, and differential entropy. In connection with maximum entropy distributions, this is the only one needed, because maximizing H ( X ) {\displaystyle H(X)} will also maximize the more general forms. The base of the logarithm is not important, as long as the same one is used consistently: Change of base merely results in a rescaling of the entropy. Information theorists may prefer to use base 2 in order to express the entropy in bits; mathematicians and physicists often prefer the natural logarithm, resulting in a unit of "nat"s for the entropy. However, the chosen measure d x {\displaystyle dx} is crucial, even though the typical use of the Lebesgue measure is often defended as a "natural" choice: Which measure is chosen determines the entropy and the consequent maximum entropy distribution.
Distributions with measured constants Many statistical distributions of applicable interest are those for which the moments or other measurable quantities are constrained to be constants. The following theorem by Ludwig Boltzmann gives the form of the probability density under these constraints.
Continuous case Suppose S {\displaystyle S} is a continuous, closed subset of the real numbers R {\displaystyle \mathbb {R} } and we choose to specify n {\displaystyle n} measurable functions f 1 , … , f n {\displaystyle f_{1},\ldots ,f_{n}} and n {\displaystyle n} numbers a 1 , … , a n . {\displaystyle a_{1},\ldots ,a_{n}.} We consider the class C {\displaystyle C} of all real-valued random variables which are supported on S {\displaystyle S} (i.e. whose density function is zero outside of S {\displaystyle S} ) and which satisfy the n {\displaystyle n} moment conditions:
E [ f j ( X ) ] ≥ a j for j = 1 , … , n {\displaystyle \operatorname {E} [f_{j}(X)]\geq a_{j}\qquad {\text{for }}\quad j=1,\ldots ,n}
… excerpt ends here. Continue reading the full article.
