Stein's lemma, named in honor of Charles Stein, is a theorem of probability theory that is of interest primarily because of its applications to statistical inference — in particular, to James–Stein estimation and empirical Bayes methods — and its applications to portfolio choice theory. The theorem gives a formula for the covariance of one random variable with the value of a function of another, when the two random variables are jointly normally distributed. Note that the name "Stein's lemma" is also commonly used to refer to a different result in the area of statistical hypothesis testing, which connects the error exponents in hypothesis testing with the Kullback–Leibler divergence. This result is also known as the Chernoff–Stein lemma and is not related to the lemma discussed in this article.
Statement Suppose X is a normally distributed random variable with expectation μ and variance σ2. Further suppose g is a differentiable function for which the two expectations E ( g ( X ) ( X − μ ) ) {\displaystyle \operatorname {E} (g(X)(X-\mu ))} and E ( g ′ ( X ) ) {\displaystyle \operatorname {E} (g'(X))} both exist. (The existence of the expectation of any random variable is equivalent to the finiteness of the expectation of its absolute value.) Then
E ( g ( X ) ( X − μ ) ) = σ 2 E ( g ′ ( X ) ) . {\displaystyle \operatorname {E} {\bigl (}g(X)(X-\mu ){\bigr )}=\sigma ^{2}\operatorname {E} {\bigl (}g'(X){\bigr )}.}
Multidimensional In general, suppose X and Y are jointly normally distributed. Then
Cov ( g ( X ) , Y ) = Cov ( X , Y ) E ( g ′ ( X ) ) . {\displaystyle \operatorname {Cov} (g(X),Y)=\operatorname {Cov} (X,Y)\operatorname {E} (g'(X)).}
For a general multivariate Gaussian random vector ( X 1 , . . . , X n ) ∼ N ( μ , Σ ) {\displaystyle (X_{1},...,X_{n})\sim {\mathcal {N}}(\mu ,\Sigma )} it follows that
E ( g ( X ) ( X − μ ) ) = Σ ⋅ E ( ∇ g ( X ) ) . {\displaystyle \operatorname {E} {\bigl (}g(X)(X-\mu ){\bigr )}=\Sigma \cdot E{\bigl (}\nabla g(X){\bigr )}.}
Similarly, when μ = 0 {\displaystyle \mu =0} , E [ ∂ i g ( X ) ] = E [ g ( X ) ( Σ − 1 X ) i ] , E [ ∂ i ∂ j g ( X ) ] = E [ g ( X ) ( ( Σ − 1 X ) i ( Σ − 1 X ) j − Σ i j − 1 ) ] {\displaystyle \operatorname {E} [\partial _{i}g(X)]=\operatorname {E} [g(X)(\Sigma ^{-1}X)_{i}],\quad \operatorname {E} [\partial _{i}\partial _{j}g(X)]=\operatorname {E} [g(X)((\Sigma ^{-1}X)_{i}(\Sigma ^{-1}X)_{j}-\Sigma _{ij}^{-1})]}
Gradient descent Stein's lemma can be used to stochastically estimate gradient: ∇ E ϵ ∼ N ( 0 , I ) ( g ( x + Σ 1 / 2 ϵ ) ) = Σ − 1 / 2 E ϵ ∼ N ( 0 , I ) ( g ( x + Σ 1 / 2 ϵ ) ϵ ) ≈ Σ − 1 / 2 1 N ∑ i = 1 N g ( x + Σ 1 / 2 ϵ i ) ϵ i {\displaystyle \nabla \operatorname {E} _{\epsilon \sim {\mathcal {N}}(0,I)}{\bigl (}g(x+\Sigma ^{1/2}\epsilon ){\bigr )}=\Sigma ^{-1/2}\operatorname {E} _{\epsilon \sim {\mathcal {N}}(0,I)}{\bigl (}g(x+\Sigma ^{1/2}\epsilon )\epsilon {\bigr )}\approx \Sigma ^{-1/2}{\frac {1}{N}}\sum _{i=1}^{N}g(x+\Sigma ^{1/2}\epsilon _{i})\epsilon _{i}} where ϵ 1 , … , ϵ N {\displaystyle \epsilon _{1},\dots ,\epsilon _{N}} are IID samples from the standard normal distribution N ( 0 , I ) {\displaystyle {\mathcal {N}}(0,I)} . This form has applications in Stein variational gradient descent and Stein variational policy gradient.
Proof The univariate probability density function for the univariate normal distribution with expectation 0 and variance 1 is
φ ( x ) = 1 2 π e − x 2 / 2 {\displaystyle \varphi (x)={1 \over {\sqrt {2\pi }}}e^{-x^{2}/2}}
Since ∫ x exp ( − x 2 / 2 ) d x = − exp ( − x 2 / 2 ) {\displaystyle \int x\exp(-x^{2}/2)\,dx=-\exp(-x^{2}/2)} we get from integration by parts:
E [ g ( X ) X ] = 1 2 π ∫ g ( x ) x exp ( − x 2 / 2 ) d x = 1 2 π ∫ g ′ ( x ) exp ( − x 2 / 2 ) d x = E [ g ′ ( X ) ] {\displaystyle \operatorname {E} [g(X)X]={\frac {1}{\sqrt {2\pi }}}\int g(x)x\exp(-x^{2}/2)\,dx={\frac {1}{\sqrt {2\pi }}}\int g'(x)\exp(-x^{2}/2)\,dx=\operatorname {E} [g'(X)]} . The case of general variance σ 2 {\displaystyle \sigma ^{2}} follows by substitution.
Generalizations Isserlis' theorem is equivalently stated as E ( X 1 f ( X 1 , … , X n ) ) = ∑ i = 1 n Cov ( X 1 , X i ) E ( ∂ X i f ( X 1 , … , X n ) ) . {\displaystyle \operatorname {E} (X_{1}f(X_{1},\ldots ,X_{n}))=\sum _{i=1}^{n}\operatorname {Cov} (X_{1},X_{i})\operatorname {E} (\partial _{X_{i}}f(X_{1},\ldots ,X_{n})).} where ( X 1 , … X n ) {\displaystyle (X_{1},\dots X_{n})} is a zero-mean multivariate normal random vector. Suppose X is in an exponential family, that is, X has the density
f η ( x ) = exp ( η ′ T ( x ) − Ψ ( η ) ) h ( x ) . {\displaystyle f_{\eta }(x)=\exp(\eta 'T(x)-\Psi (\eta ))h(x).}
Suppose this density has support ( a , b ) {\displaystyle (a,b)} where a , b {\displaystyle a,b} could be − ∞ , ∞ {\displaystyle -\infty ,\infty } and as x → a or b {\displaystyle x\rightarrow a{\text{ or }}b} , exp ( η ′ T ( x ) ) h ( x ) g ( x ) → 0 {\displaystyle \exp(\eta 'T(x))h(x)g(x)\rightarrow 0} where g {\displaystyle g} is any differentiable function such that E | g ′ ( X ) | < ∞ {\displaystyle E|g'(X)|<\infty } or exp ( η ′ T ( x ) ) h ( x ) → 0 {\displaystyle \exp(\eta 'T(x))h(x)\rightarrow 0} if a , b {\displaystyle a,b} finite. Then
E [ ( h ′ ( X ) h ( X ) + ∑ η i T i ′ ( X ) ) ⋅ g ( X ) ] = − E [ g ′ ( X ) ] . {\displaystyle E\left[\left({\frac {h'(X)}{h(X)}}+\sum \eta _{i}T_{i}'(X)\right)\cdot g(X)\right]=-E[g'(X)].}
The derivation is same as the special case, namely, integration by parts. If we only know X {\displaystyle X} has support R {\displaystyle \mathbb {R} } , then it could be the case that E | g ( X ) | < ∞ and E | g ′ ( X ) | < ∞ {\displaystyle E|g(X)|<\infty {\text{ and }}E|g'(X)|<\infty } but lim x → ∞ f η ( x ) g ( x ) ≠ 0 {\displaystyle \lim _{x\rightarrow \infty }f_{\eta }(x)g(x)\not =0} . To see this, simply put g ( x ) = 1 {\displaystyle g(x)=1} and f η ( x ) {\displaystyle f_{\eta }(x)} with infinitely spikes towards infinity but still integrable. One such example could be adapted from f ( x ) = { 1 x ∈ [ n , n + 2 − n ) 0 otherwise {\displaystyle f(x)={\begin{cases}1&x\in [n,n+2^{-n})\\0&{\text{otherwise}}\end{cases}}} so that f {\displaystyle f} is smooth. Extensions to elliptically-contoured distributions also exist.
See also Stein's method Taylor expansions for the moments of functions of random variables Stein discrepancy
References