In mathematics, Jensen's inequality, named after the Danish mathematician Johan Jensen, relates the value of a convex function of an integral to the integral of the convex function. It was proved by Jensen in 1906, building on an earlier proof of the same inequality for doubly-differentiable functions by Otto Hölder in 1889. Given its generality, the inequality appears in many forms depending on the context, some of which are presented below. In its simplest form the inequality states that the convex transformation of a mean is less than or equal to the mean applied after convex transformation (or equivalently, the opposite inequality for concave transformations). Jensen's inequality generalizes the statement that the secant line of a convex function lies above the graph of the function, which is Jensen's inequality for two points: the secant line consists of weighted means of the convex function (for t ∈ [0,1]),
t f ( x 1 ) + ( 1 − t ) f ( x 2 ) , {\displaystyle tf(x_{1})+(1-t)f(x_{2}),}
while the graph of the function is the convex function of the weighted means,
f ( t x 1 + ( 1 − t ) x 2 ) . {\displaystyle f(tx_{1}+(1-t)x_{2}).}
Thus, Jensen's inequality in this case is
f ( t x 1 + ( 1 − t ) x 2 ) ≤ t f ( x 1 ) + ( 1 − t ) f ( x 2 ) . {\displaystyle f(tx_{1}+(1-t)x_{2})\leq tf(x_{1})+(1-t)f(x_{2}).}
In the context of probability theory, it is generally stated in the following form: if X is a random variable and φ is a convex function, then
φ ( E [ X ] ) ≤ E [ φ ( X ) ] . {\displaystyle \varphi (\operatorname {E} [X])\leq \operatorname {E} \left[\varphi (X)\right].}
The difference between the two sides of the inequality, E [ φ ( X ) ] − φ ( E [ X ] ) {\displaystyle \operatorname {E} \left[\varphi (X)\right]-\varphi \left(\operatorname {E} [X]\right)} , is called the Jensen gap.
Statements The classical form of Jensen's inequality involves several numbers and weights. The inequality can be stated quite generally using either the language of measure theory or (equivalently) probability. In the probabilistic setting, the inequality can be further generalized to its full strength.
Finite form For a real convex function φ {\displaystyle \varphi } , numbers x 1 , x 2 , … , x n {\displaystyle x_{1},x_{2},\ldots ,x_{n}} in its domain, and positive weights a i {\displaystyle a_{i}} , Jensen's inequality can be stated as:
and the inequality is reversed if φ {\displaystyle \varphi } is concave, which is
Equality holds if and only if x 1 = x 2 = ⋯ = x n {\displaystyle x_{1}=x_{2}=\cdots =x_{n}} or φ {\displaystyle \varphi } is linear on a domain containing x 1 , x 2 , ⋯ , x n {\displaystyle x_{1},x_{2},\cdots ,x_{n}} . As a particular case, if the weights a i {\displaystyle a_{i}} are all equal, then (1) and (2) become
For instance, the function log(x) is concave, so substituting φ ( x ) = log ( x ) {\displaystyle \varphi (x)=\log(x)} in the previous formula (4) establishes the (logarithm of the) familiar arithmetic-mean/geometric-mean inequality:
log ( ∑ i = 1 n x i n ) ≥ ∑ i = 1 n log ( x i ) n {\displaystyle \log \!\left({\frac {\sum _{i=1}^{n}x_{i}}{n}}\right)\geq {\frac {\sum _{i=1}^{n}\log \!\left(x_{i}\right)}{n}}}
exp ( log ( ∑ i = 1 n x i n ) ) ≥ exp ( ∑ i = 1 n log ( x i ) n ) {\displaystyle \exp \!\left(\log \!\left({\frac {\sum _{i=1}^{n}x_{i}}{n}}\right)\right)\geq \exp \!\left({\frac {\sum _{i=1}^{n}\log \!\left(x_{i}\right)}{n}}\right)}
x 1 + x 2 + ⋯ + x n n ≥ x 1 ⋅ x 2 ⋯ x n n {\displaystyle {\frac {x_{1}+x_{2}+\cdots +x_{n}}{n}}\geq {\sqrt[{n}]{x_{1}\cdot x_{2}\cdots x_{n}}}}
A common application has x as a function of another variable (or set of variables) t, that is, x i = g ( t i ) {\displaystyle x_{i}=g(t_{i})} . All of this carries directly over to the general continuous case: the weights ai are replaced by a non-negative integrable function f (x), such as a probability distribution, and the summations are replaced by integrals.
Measure-theoretic form Let ( Ω , A , μ ) {\displaystyle (\Omega ,A,\mu )} be a probability space. Let f : Ω → R {\displaystyle f:\Omega \to \mathbb {R} } be a μ {\displaystyle \mu } -integrable function and φ : R → R {\displaystyle \varphi :\mathbb {R} \to \mathbb {R} } be convex. Then:
φ ( ∫ Ω f d μ ) ≤ ∫ Ω φ ∘ f d μ {\displaystyle \varphi \left(\int _{\Omega }f\,\mathrm {d} \mu \right)\leq \int _{\Omega }\varphi \circ f\,\mathrm {d} \mu }
In real analysis, we may require an estimate on
φ ( ∫ a b f ( x ) d x ) {\displaystyle \varphi \left(\int _{a}^{b}f(x)\,dx\right)}
where a , b ∈ R {\displaystyle a,b\in \mathbb {R} } , and f : [ a , b ] → R {\displaystyle f\colon [a,b]\to \mathbb {R} } is a non-negative Lebesgue-integrable function. In this case, the Lebesgue measure of [ a , b ] {\displaystyle [a,b]} need not be 1. However, by integration by substitution, the interval can be rescaled so that it has measure 1. Then Jensen's inequality can be applied to get
φ ( 1 b − a ∫ a b f ( x ) d x ) ≤ 1 b − a ∫ a b φ ( f ( x ) ) d x . {\displaystyle \varphi \left({\frac {1}{b-a}}\int _{a}^{b}f(x)\,dx\right)\leq {\frac {1}{b-a}}\int _{a}^{b}\varphi (f(x))\,dx.}
Probabilistic form The same result can be equivalently stated in a probability theory setting, by a simple change of notation. Let ( Ω , F , P ) {\displaystyle (\Omega ,{\mathfrak {F}},\operatorname {P} )} be a probability space, X an integrable real-valued random variable and φ {\displaystyle \varphi } a convex function. Then
φ ( E [ X ] ) ≤ E [ φ ( X ) ] . {\displaystyle \varphi {\big (}\operatorname {E} [X]{\big )}\leq \operatorname {E} [\varphi (X)].}
In this probability setting, the measure μ is intended as a probability P {\displaystyle \operatorname {P} } , the integral with respect to μ as an expected value E {\displaystyle \operatorname {E} } , and the function f {\displaystyle f} as a random variable X. Note that the equality holds if and only if φ {\displaystyle \varphi } is a linear function on some convex set A {\displaystyle A} such that P ( X ∈ A ) = 1 {\displaystyle P(X\in A)=1} (which follows by inspecting the measure-theoretical proof below).
General inequality in a probabilistic setting More generally, let T be a real topological vector space, and X a T-valued integrable random variable. In this general setting, integrable means that there exists an element E [ X ] {\displaystyle \operatorname {E} [X]} in T, such that for any element z in the dual space of T: E | ⟨ z , X ⟩ | < ∞ {\displaystyle \operatorname {E} |\langle z,X\rangle |<\infty } , and ⟨ z , E [ X ] ⟩ = E [ ⟨ z , X ⟩ ] {\displaystyle \langle z,\operatorname {E} [X]\rangle =\operatorname {E} [\langle z,X\rangle ]} . Then, for any measurable convex function φ and any sub-σ-algebra G {\displaystyle {\mathfrak {G}}} of F {\displaystyle {\mathfrak {F}}} :
φ ( E [ X ∣ G ] ) ≤ E [ φ ( X ) ∣ G ] . {\displaystyle \varphi \left(\operatorname {E} \left[X\mid {\mathfrak {G}}\right]\right)\leq \operatorname {E} \left[\varphi (X)\mid {\mathfrak {G}}\right].}
Here E [ ⋅ ∣ G ] {\displaystyle \operatorname {E} [\cdot \mid {\mathfrak {G}}]} stands for the expectation conditioned to the σ-algebra G {\displaystyle {\mathfrak {G}}} . This general statement reduces to the previous ones when the topological vector space T is the real axis, and G {\displaystyle {\mathfrak {G}}} is the trivial σ-algebra {∅, Ω} (where ∅ is the empty set, and Ω is the sample space).
A sharpened and generalized form Let X be a one-dimensional random variable with mean μ {\displaystyle \mu } and variance σ 2 ≥ 0 {\displaystyle \sigma ^{2}\geq 0} . Let φ ( x ) {\displaystyle \varphi (x)} be a twice differentiable function, and define the function
h ( x ) ≜ φ ( x ) − φ ( μ ) ( x − μ ) 2 − φ ′ ( μ ) x − μ . {\displaystyle h(x)\triangleq {\frac {\varphi \left(x\right)-\varphi \left(\mu \right)}{\left(x-\mu \right)^{2}}}-{\frac {\varphi '\left(\mu \right)}{x-\mu }}.}
Then
σ 2 inf φ ″ ( x ) 2 ≤ σ 2 inf h ( x ) ≤ E [ φ ( X ) ] − φ ( E [ X ] ) ≤ σ 2 sup h ( x ) ≤ σ 2 sup φ ″ ( x ) 2 . {\displaystyle \sigma ^{2}\inf {\frac {\varphi ''(x)}{2}}\leq \sigma ^{2}\inf h(x)\leq E\left[\varphi \left(X\right)\right]-\varphi \left(E[X]\right)\leq \sigma ^{2}\sup h(x)\leq \sigma ^{2}\sup {\frac {\varphi ''(x)}{2}}.}
In particular, when φ ( x ) {\displaystyle \varphi (x)} is convex, then φ ″ ( x ) ≥ 0 {\displaystyle \varphi ''(x)\geq 0} , and the standard form of Jensen's inequality immediately follows for the case where φ ( x ) {\displaystyle \varphi (x)} is additionally assumed to be twice differentiable.
Proofs
Intuitive graphical proof
Jensen's inequality can be proved in several ways, and three different proofs corresponding to the different statements above will be offered. Before embarking on these mathematical derivations, however, it is worth analyzing an intuitive graphical argument based on the probabilistic case where X is a real number (see figure). Assuming a hypothetical distribution of X values, one can immediately identify the position of E [ X ] {\displaystyle \operatorname {E} [X]} and its image φ ( E [ X ] ) {\displaystyle \varphi (\operatorname {E} [X])} in the graph. Noticing that for convex mappings Y = φ(x) of some x values the corresponding distribution of Y values is increasingly "stretched up" for increasing values of X, it is easy to see that the distribution of Y is broader in the interval corresponding to X > X0 and narrower in X < X0 for any X0; in particular, this is also true for X 0 = E [ X ] {\displaystyle X_{0}=\operatorname {E} [X]} . Consequently, in this picture the expectation of Y will always shift upwards with respect to the position of φ ( E [ X ] ) {\displaystyle \varphi (\operatorname {E} [X])} . A similar reasoning holds if the distribution of X covers a decreasing portion of the convex function, or both a decreasing and an increasing portion of it. This "proves" the inequality, i.e.
φ ( E [ X ] ) ≤ E [ φ ( X ) ] = E [ Y ] , {\displaystyle \varphi (\operatorname {E} [X])\leq \operatorname {E} [\varphi (X)]=\operatorname {E} [Y],}
with equality when φ(X) is not strictly convex, e.g. when it is a straight line, or when X follows a degenerate distribution (i.e. is a constant). The proofs below formalize this intuitive notion.
Proof 1 (finite form) If λ1 and λ2 are two arbitrary nonnegative real numbers such that λ1 + λ2 = 1 then convexity of φ implies
∀ x 1 , x 2 : φ ( λ 1 x 1 + λ 2 x 2 ) ≤ λ 1 φ ( x 1 ) + λ 2 φ ( x 2 ) . {\displaystyle \forall x_{1},x_{2}:\qquad \varphi \left(\lambda _{1}x_{1}+\lambda _{2}x_{2}\right)\leq \lambda _{1}\,\varphi (x_{1})+\lambda _{2}\,\varphi (x_{2}).}
This can be generalized: if λ1, ..., λn are nonnegative real numbers such that λ1 + ... + λn = 1, then
φ ( λ 1 x 1 + λ 2 x 2 + ⋯ + λ n x n ) ≤ λ 1 φ ( x 1 ) + λ 2 φ ( x 2 ) + ⋯ + λ n φ ( x n ) , {\displaystyle \varphi (\lambda _{1}x_{1}+\lambda _{2}x_{2}+\cdots +\lambda _{n}x_{n})\leq \lambda _{1}\,\varphi (x_{1})+\lambda _{2}\,\varphi (x_{2})+\cdots +\lambda _{n}\,\varphi (x_{n}),}
for any x1, ..., xn. The finite form of the Jensen's inequality can be proved by induction: by convexity hypotheses, the statement is true for n = 2. Suppose the statement is true for some n, so
φ ( ∑ i = 1 n λ i x i ) ≤ ∑ i = 1 n λ i φ ( x i ) {\displaystyle \varphi \left(\sum _{i=1}^{n}\lambda _{i}x_{i}\right)\leq \sum _{i=1}^{n}\lambda _{i}\varphi \left(x_{i}\right)}
for any λ1, ..., λn such that λ1 + ... + λn = 1. One needs to prove it for n + 1. At least one of the λi is strictly smaller than 1 {\displaystyle 1} , say λn+1; therefore by convexity inequality:
φ ( ∑ i = 1 n + 1 λ i x i ) = φ ( ( 1 − λ n + 1 ) ∑ i = 1 n λ i 1 − λ n + 1 x i + λ n + 1 x n + 1 ) ≤ ( 1 − λ n + 1 ) φ ( ∑ i = 1 n λ i 1 − λ n + 1 x i ) + λ n + 1 φ ( x n + 1 ) . {\displaystyle {\begin{aligned}\varphi \left(\sum _{i=1}^{n+1}\lambda _{i}x_{i}\right)&=\varphi \left((1-\lambda _{n+1})\sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}x_{i}+\lambda _{n+1}x_{n+1}\right)\\&\leq (1-\lambda _{n+1})\varphi \left(\sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}x_{i}\right)+\lambda _{n+1}\,\varphi (x_{n+1}).\end{aligned}}}
Since λ1 + ... +λn + λn+1 = 1,
∑ i = 1 n λ i 1 − λ n + 1 = 1 {\displaystyle \sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}=1} , applying the inductive hypothesis gives
φ ( ∑ i = 1 n λ i 1 − λ n + 1 x i ) ≤ ∑ i = 1 n λ i 1 − λ n + 1 φ ( x i ) {\displaystyle \varphi \left(\sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}x_{i}\right)\leq \sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}\varphi (x_{i})}
therefore
φ ( ∑ i = 1 n + 1 λ i x i ) ≤ ( 1 − λ n + 1 ) ∑ i = 1 n λ i 1 − λ n + 1 φ ( x i ) + λ n + 1 φ ( x n + 1 ) = ∑ i = 1 n + 1 λ i φ ( x i ) {\displaystyle {\begin{aligned}\varphi \left(\sum _{i=1}^{n+1}\lambda _{i}x_{i}\right)&\leq (1-\lambda _{n+1})\sum _{i=1}^{n}{\frac {\lambda _{i}}{1-\lambda _{n+1}}}\varphi (x_{i})+\lambda _{n+1}\,\varphi (x_{n+1})=\sum _{i=1}^{n+1}\lambda _{i}\varphi (x_{i})\end{aligned}}}
We deduce the inequality is true for n + 1, by induction it follows that the result is also true for all integer n greater than 2. In order to obtain the general inequality from this finite form, one needs to use a density argument. The finite form can be rewritten as:
φ ( ∫ x d μ n ( x ) ) ≤ ∫ φ ( x ) d μ n ( x ) , {\displaystyle \varphi \left(\int x\,d\mu _{n}(x)\right)\leq \int \varphi (x)\,d\mu _{n}(x),}
where μn is a measure given by an arbitrary convex combination of Dirac deltas:
μ n = ∑ i = 1 n λ i δ x i . {\displaystyle \mu _{n}=\sum _{i=1}^{n}\lambda _{i}\delta _{x_{i}}.}
Since convex functions are continuous, and since convex combinations of Dirac deltas are weakly dense in the set of probability measures (as could be easily verified), the general statement is obtained simply by a limiting procedure.
Proof 2 (measure-theoretic form) Let