Non-linear least squares is the form of least squares analysis used to fit a set of m observations with a model that is non-linear in n unknown parameters (m ≥ n). It is used in some forms of nonlinear regression. The basis of the method is to approximate the model by a linear one and to refine the parameters by successive iterations. There are many similarities to linear least squares, but also some significant differences. In economic theory, the non-linear least squares method is applied in (i) the probit regression, (ii) threshold regression, (iii) smooth regression, (iv) logistic link regression, (v) Box–Cox transformed regressors ( m ( x , θ i ) = θ 1 + θ 2 x ( θ 3 ) {\displaystyle m(x,\theta _{i})=\theta _{1}+\theta _{2}x^{(\theta _{3})}} ).
Theory Consider a set of m {\displaystyle m} data points, ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x m , y m ) , {\displaystyle (x_{1},y_{1}),(x_{2},y_{2}),\dots ,(x_{m},y_{m}),} and a curve (model function) y ^ = f ( x , β ) , {\displaystyle {\hat {y}}=f(x,{\boldsymbol {\beta }}),} that in addition to the variable x {\displaystyle x} also depends on n {\displaystyle n} parameters, β = ( β 1 , β 2 , … , β n ) , {\displaystyle {\boldsymbol {\beta }}=(\beta _{1},\beta _{2},\dots ,\beta _{n}),} with m ≥ n . {\displaystyle m\geq n.} It is desired to find the vector β {\displaystyle {\boldsymbol {\beta }}} of parameters such that the curve fits best the given data in the least squares sense, that is, the sum of squares
S = ∑ i = 1 m r i 2 {\displaystyle S=\sum _{i=1}^{m}r_{i}^{2}}
is minimized, where the residuals (in-sample prediction errors) ri are given by
r i = y i − f ( x i , β ) {\displaystyle r_{i}=y_{i}-f(x_{i},{\boldsymbol {\beta }})}
for i = 1 , 2 , … , m . {\displaystyle i=1,2,\dots ,m.}
The minimum value of S occurs when the gradient is zero. Since the model contains n parameters there are n gradient equations:
∂ S ∂ β j = 2 ∑ i r i ∂ r i ∂ β j = 0 ( j = 1 , … , n ) . {\displaystyle {\frac {\partial S}{\partial \beta _{j}}}=2\sum _{i}r_{i}{\frac {\partial r_{i}}{\partial \beta _{j}}}=0\quad (j=1,\ldots ,n).}
In a nonlinear system, the derivatives ∂ r i ∂ β j {\textstyle {\frac {\partial r_{i}}{\partial \beta _{j}}}} are functions of both the independent variable and the parameters, so in general these gradient equations do not have a closed solution. Instead, initial values must be chosen for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation,
… excerpt ends here. Continue reading the full article.
