In probability theory, Hoeffding's lemma is an inequality that bounds the moment-generating function of any bounded random variable, implying that such variables are subgaussian. It is named after the Finnish–American mathematical statistician Wassily Hoeffding. The proof of Hoeffding's lemma uses Taylor's theorem. Hoeffding's lemma is itself used in the proof of Hoeffding's inequality as well as the generalization McDiarmid's inequality.
Statement Let X be any real-valued random variable such that a ≤ X ≤ b {\displaystyle a\leq X\leq b} almost surely, i.e. with probability one. Then, for all λ ∈ R {\displaystyle \lambda \in \mathbb {R} } ,
E [ e λ X ] ≤ exp ( λ E [ X ] + λ 2 ( b − a ) 2 8 ) , {\displaystyle \mathbb {E} \left[e^{\lambda X}\right]\leq \exp {\Big (}\lambda \mathbb {E} [X]+{\frac {\lambda ^{2}(b-a)^{2}}{8}}{\Big )},}
or equivalently,
E [ e λ ( X − E [ X ] ) ] ≤ exp ( λ 2 ( b − a ) 2 8 ) . {\displaystyle \mathbb {E} \left[e^{\lambda (X-\mathbb {E} [X])}\right]\leq \exp {\Big (}{\frac {\lambda ^{2}(b-a)^{2}}{8}}{\Big )}.}
Proof The following proof is direct but somewhat ad-hoc.
Statement This statement and proof uses the language of subgaussian variables and exponential tilting, and is less ad-hoc. Let X {\displaystyle X} be any real-valued random variable such that a ≤ X ≤ b {\displaystyle a\leq X\leq b} almost surely, i.e. with probability one. Then it is subgaussian with variance proxy norm ‖ X ‖ v p ≤ b − a 2 {\displaystyle \|X\|_{vp}\leq {\frac {b-a}{2}}} .
Given this general case, the formula E [ e λ ( X − E [ X ] ) ] ≤ e λ 2 ( b − a ) 2 8 {\displaystyle \mathbb {E} \left[e^{\lambda (X-\mathbb {E} [X])}\right]\leq e^{\frac {\lambda ^{2}(b-a)^{2}}{8}}} is a mere corollary of a general property of variance proxy.
See also Hoeffding's inequality Bennett's inequality
Notes