Weighted least squares (WLS), also known as weighted linear regression, is a generalization of ordinary least squares and linear regression in which knowledge of the unequal variance of observations (heteroscedasticity) is incorporated into the regression. WLS is also a specialization of generalized least squares, when all the off-diagonal entries of the covariance matrix of the errors are null.
Formulation The fit of a model to a data point is measured by its residual, r i {\displaystyle r_{i}} , defined as the difference between a measured value of the dependent variable, y i {\displaystyle y_{i}} and the value predicted by the model, f ( x i , β ) {\displaystyle f(x_{i},{\boldsymbol {\beta }})} :
r i ( β ) = y i − f ( x i , β ) . {\displaystyle r_{i}({\boldsymbol {\beta }})=y_{i}-f(x_{i},{\boldsymbol {\beta }}).}
If the errors are uncorrelated and have equal variance, then the function
S ( β ) = ∑ i r i ( β ) 2 , {\displaystyle S({\boldsymbol {\beta }})=\sum _{i}r_{i}({\boldsymbol {\beta }})^{2},}
is minimized at β ^ {\displaystyle {\boldsymbol {\hat {\beta }}}} , such that ∂ S ∂ β j ( β ^ ) = 0 {\displaystyle {\frac {\partial S}{\partial \beta _{j}}}({\hat {\boldsymbol {\beta }}})=0} . The Gauss–Markov theorem shows that, when this is so, β ^ {\displaystyle {\hat {\boldsymbol {\beta }}}} is a best linear unbiased estimator (BLUE). If, however, the measurements are uncorrelated but have different uncertainties, a modified approach might be adopted. Aitken showed that when a weighted sum of squared residuals is minimized, β ^ {\displaystyle {\hat {\boldsymbol {\beta }}}} is the BLUE if each weight is equal to the reciprocal of the variance of the measurement
S = ∑ i = 1 n W i i r i 2 , W i i = 1 σ i 2 {\displaystyle {\begin{aligned}S&=\sum _{i=1}^{n}W_{ii}{r_{i}}^{2},&W_{ii}&={\frac {1}{{\sigma _{i}}^{2}}}\end{aligned}}}
The gradient equations for this sum of squares are
− 2 ∑ i W i i ∂ f ( x i , β ) ∂ β j r i = 0 , j = 1 , … , m {\displaystyle -2\sum _{i}W_{ii}{\frac {\partial f(x_{i},{\boldsymbol {\beta }})}{\partial \beta _{j}}}r_{i}=0,\quad j=1,\ldots ,m}
which, in a linear least squares system give the modified normal equations,
… excerpt ends here. Continue reading the full article.
