In statistics, kernel regression is a non-parametric technique to estimate the conditional expectation of a random variable. The objective is to find a non-linear relation between a pair of random variables X and Y. In any nonparametric regression, the conditional expectation of a variable Y {\displaystyle Y} relative to a variable X {\displaystyle X} may be written:
E ( Y ∣ X ) = m ( X ) {\displaystyle \operatorname {E} (Y\mid X)=m(X)}
where m {\displaystyle m} is an unknown function.
Nadaraya–Watson kernel regression Nadaraya and Watson, both in 1964, proposed to estimate m {\displaystyle m} as a locally weighted average, using a kernel as a weighting function. The Nadaraya–Watson estimator is:
m ^ h ( x ) = ∑ i = 1 n K h ( x − x i ) y i ∑ i = 1 n K h ( x − x i ) {\displaystyle {\widehat {m}}_{h}(x)={\frac {\sum _{i=1}^{n}K_{h}(x-x_{i})y_{i}}{\sum _{i=1}^{n}K_{h}(x-x_{i})}}}
where K h ( t ) = 1 h K ( t h ) {\displaystyle K_{h}(t)={\frac {1}{h}}K\left({\frac {t}{h}}\right)} is a kernel with a bandwidth h {\displaystyle h} such that K ( ⋅ ) {\displaystyle K(\cdot )} is of order at least 1, that is ∫ − ∞ ∞ u K ( u ) d u = 0 {\displaystyle \int _{-\infty }^{\infty }uK(u)\,du=0} .
Derivation Starting with the definition of conditional expectation,
E ( Y ∣ X = x ) = ∫ y f ( y ∣ x ) d y = ∫ y f ( x , y ) f ( x ) d y {\displaystyle \operatorname {E} (Y\mid X=x)=\int yf(y\mid x)\,dy=\int y{\frac {f(x,y)}{f(x)}}\,dy}
we estimate the joint distributions f(x,y) and f(x) using kernel density estimation with a kernel K:
f ^ ( x , y ) = 1 n ∑ i = 1 n K h ( x − x i ) K h ( y − y i ) , {\displaystyle {\hat {f}}(x,y)={\frac {1}{n}}\sum _{i=1}^{n}K_{h}(x-x_{i})K_{h}(y-y_{i}),}
f ^ ( x ) = 1 n ∑ i = 1 n K h ( x − x i ) , {\displaystyle {\hat {f}}(x)={\frac {1}{n}}\sum _{i=1}^{n}K_{h}(x-x_{i}),}
We get:
… excerpt ends here. Continue reading the full article.
