A Stein discrepancy is a statistical divergence between two probability measures that is rooted in Stein's method. It was first formulated as a tool to assess the quality of Markov chain Monte Carlo samplers, but has since been used in diverse settings in statistics, machine learning and computer science.
Definition Let X {\displaystyle {\mathcal {X}}} be a measurable space and let M {\displaystyle {\mathcal {M}}} be a set of measurable functions of the form m : X → R {\displaystyle m:{\mathcal {X}}\rightarrow \mathbb {R} } . A natural notion of distance between two probability distributions P {\displaystyle P} , Q {\displaystyle Q} , defined on X {\displaystyle {\mathcal {X}}} , is provided by an integral probability metric
( 1.1 ) d M ( P , Q ) := sup m ∈ M | E X ∼ P [ m ( X ) ] − E Y ∼ Q [ m ( Y ) ] | , {\displaystyle (1.1)\quad d_{\mathcal {M}}(P,Q):=\sup _{m\in {\mathcal {M}}}|\mathbb {E} _{X\sim P}[m(X)]-\mathbb {E} _{Y\sim Q}[m(Y)]|,}
where for the purposes of exposition we assume that the expectations exist, and that the set M {\displaystyle {\mathcal {M}}} is sufficiently rich that (1.1) is indeed a metric on the set of probability distributions on X {\displaystyle {\mathcal {X}}} , i.e. d M ( P , Q ) = 0 {\displaystyle d_{\mathcal {M}}(P,Q)=0} if and only if P = Q {\displaystyle P=Q} . The choice of the set M {\displaystyle {\mathcal {M}}} determines the topological properties of (1.1). However, for practical purposes the evaluation of (1.1) requires access to both P {\displaystyle P} and Q {\displaystyle Q} , often rendering direct computation of (1.1) impractical. Stein's method is a theoretical tool that can be used to bound (1.1). Specifically, we suppose that we can identify an operator A P {\displaystyle {\mathcal {A}}_{P}} and a set F P {\displaystyle {\mathcal {F}}_{P}} of real-valued functions in the domain of A P {\displaystyle {\mathcal {A}}_{P}} , both of which may be P {\displaystyle P} -dependent, such that for each m ∈ M {\displaystyle m\in {\mathcal {M}}} there exists a solution f m ∈ F P {\displaystyle f_{m}\in {\mathcal {F}}_{P}} to the Stein equation
( 1.2 ) m ( x ) − E X ∼ P [ m ( X ) ] = A P f m ( x ) . {\displaystyle (1.2)\quad m(x)-\mathbb {E} _{X\sim P}[m(X)]={\mathcal {A}}_{P}f_{m}(x).}
The operator A P {\displaystyle {\mathcal {A}}_{P}} is termed a Stein operator and the set F P {\displaystyle {\mathcal {F}}_{P}} is called a Stein set. Substituting (1.2) into (1.1), we obtain an upper bound
… excerpt ends here. Continue reading the full article.
