Ridge regression (also known as Tikhonov regularization, named for Andrey Tikhonov) is a method of estimating the coefficients of multiple-regression models in scenarios where the variables are highly correlated. It has been used in many fields including econometrics, chemistry, and engineering. It is a widely used method of regularization of ill-posed problem inverse problems. It is particularly useful to mitigate the problem of multicollinearity in linear regression, which commonly occurs in models with large numbers of parameters. In general, the method provides improved efficiency in parameter estimation problems in exchange for a tolerable amount of bias (see bias–variance tradeoff). The theory was first introduced by Hoerl and Kennard in 1970 in their Technometrics papers "Ridge regressions: biased estimation of nonorthogonal problems" and "Ridge regressions: applications in nonorthogonal problems". Ridge regression was developed as a possible solution to the imprecision of least square estimators when linear regression models have some multicollinear (highly correlated) independent variables—by creating a ridge regression estimator (RR). This provides a more precise ridge parameters estimate, as its variance and mean square estimator are often smaller than the least square estimators previously derived.
Overview In the ordinary least squares solution of
Y = X β + ε , {\displaystyle \mathbf {Y} =\mathbf {X} {\boldsymbol {\beta }}+{\boldsymbol {\varepsilon }},\,}
the problem of a near-singular moment matrix X T X {\displaystyle \mathbf {X} ^{\mathsf {T}}\mathbf {X} } is alleviated by adding positive elements to the diagonals, thereby decreasing its condition number. Compared to the ordinary least squares estimator, the simple ridge estimator has an extra term λ I {\displaystyle \lambda \mathbf {I} } in the denominator:
β ^ λ = ( X T X + λ I ) − 1 X T Y {\displaystyle {\hat {\boldsymbol {\beta }}}_{\lambda }=\left(\mathbf {X} ^{\mathsf {T}}\mathbf {X} +\lambda \mathbf {I} \right)^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {Y} }
where Y {\displaystyle \mathbf {Y} } is the regressand or response vector, X {\displaystyle \mathbf {X} } is the design matrix, I {\displaystyle \mathbf {I} } is the identity matrix, and the ridge (or Tikhonov) regularization parameter λ ≥ 0 {\displaystyle \lambda \geq 0} serves as the constant shifting the diagonals of the moment matrix. It can be shown that this estimator is the solution to the least squares problem subject to the constraint β T β = c {\displaystyle {\boldsymbol {\beta }}^{\mathsf {T}}{\boldsymbol {\beta }}=c} , which can be expressed as a Lagrangian minimization:
argmin β ‖ Y − X β ‖ 2 + λ ( β T β − c ) {\displaystyle {\text{argmin}}_{\boldsymbol {\beta }}\,\|\mathbf {Y} -\mathbf {X} {\boldsymbol {\beta }}\|^{2}+\lambda \left({\boldsymbol {\beta }}^{\mathsf {T}}{\boldsymbol {\beta }}-c\right)}
which shows that λ {\displaystyle \lambda } is nothing but the Lagrange multiplier of the constraint. In fact, there is a one-to-one relationship between c {\displaystyle c} and λ {\displaystyle \lambda } and since, in practice, we do not know c {\displaystyle c} , we define λ {\displaystyle \lambda } heuristically or find it via additional data-fitting strategies, see Determination of the Tikhonov parameter below. Note that as λ ↓ 0 {\displaystyle \lambda \downarrow 0} , the constraint eventually becomes non-binding, and the ridge estimator converges to the minimum-norm ordinary least squares estimator, here denoted as β ^ = β ^ 0 {\displaystyle {\hat {\boldsymbol {\beta }}}={\hat {\boldsymbol {\beta }}}_{0}} :
… excerpt ends here. Continue reading the full article.
