In statistics and econometrics, the Frisch–Waugh–Lovell (FWL) theorem is a theorem of ordinary least squares estimators which states that estimates of multiple regression coefficients depend only on the unique variation of each explanatory variable. The theorem is named for econometricians Ragnar Frisch, Frederick V. Waugh, and Michael C. Lovell. Ordinary least squares is a method of estimating coefficients in a linear regression where a single dependent variable is modeled as a linear function of one or more explanatory variables. The Frisch–Waugh–Lovell theorem states that, in a linear regression model fitted by least squares, each explanatory variable's coefficient reflects the relationship between the dependent variable and the part of that explanatory variable which is not linearly related to the other explanatory variables. Specifically, each explanatory variable can be decomposed into two parts: the part linearly related to the other explanatory variables, and a residual component. Then, that variable's coefficient can be found by regressing the dependent variable on the residual component. As a result, the theorem is sometimes called the regression anatomy theorem. Coefficients in least squares-estimated regressions are often interpreted as the effect of the respective variable controlling for or holding constant the set of other explanatory variables. However, given 'high' multicollinearity, "Interpreting coefficients [...] should be carried out with caution". The Frisch–Waugh–Lovell theorem shows that coefficients represent effects due to unique variation in each explanatory variable. An initial version of the theorem was introduced by Udny Yule in 1907, though it was not popularized in economics until a 1933 paper by Ragnar Frisch and Frederick Waugh in the first volume of Econometrica. At the time, there was debate among economists over the proper way to adjust statistical models for the effect of time trends. Frisch and Waugh used the theorem to show that the two leading methods of adjustment were numerically equivalent, resolving the debate. Michael Lovell contributed to the theorem's development through a 1963 paper generalizing Frisch and Waugh's result beyond time trends to arbitrary sets of explanatory variables.
Background The Frisch–Waugh–Lovell theorem is a result for regressions estimated by ordinary least squares, the most commonly used estimator in applied econometrics. A regression is a statistical model where a dependent variable is modeled as a function of one or more explanatory variables plus some residual term. Ordinary least squares can be used when a dependent variable is modeled as a linear combination of one or more explanatory variables plus some residual term. For example, an individual's wages may be modeled as a linear function of a constant term, education, and parental income, with a residual term that encompasses deviations from the model's prediction. Ordinary least squares sets the values of the coefficients to minimize the sum of squared residuals. Under a certain set of assumptions, the hypotheses of the Gauss–Markov theorem, least squares estimation is the best linear unbiased estimator. Let y {\displaystyle y} be any dependent variable and x 1 , x 2 , … , x k {\displaystyle x_{1},\,x_{2},\,\dots ,\,x_{k}} a set of k {\displaystyle k} explanatory variables, and suppose n {\displaystyle n} observations of ( y i , x 1 i , x 2 i , … , x k i ) {\displaystyle (y_{i},x_{1i},x_{2i},\dots ,x_{ki})} are obtained. If y {\displaystyle y} is modeled as a linear function of the explanatory variables and a constant, the estimated model can be written as y i = β ^ 0 + β ^ 1 x 1 i + β ^ 2 x 2 i + ⋯ + β ^ k x k i + e ^ i {\displaystyle y_{i}={\hat {\beta }}_{0}+{\hat {\beta }}_{1}x_{1i}+{\hat {\beta }}_{2}x_{2i}+\cdots +{\hat {\beta }}_{k}x_{ki}+{\hat {e}}_{i}} , where the hats denote estimates of the respective parameters. The least squares estimator sets the coefficients β ^ j , j ∈ { 0 , … , k } {\displaystyle {\hat {\beta }}_{j},j\in \{0,\dots ,k\}} to minimize the sum of squared residuals ∑ i = 1 n e ^ i 2 {\textstyle \sum _{i=1}^{n}{\hat {e}}_{i}^{2}} . With n {\displaystyle n} observations this involves minimizing across n {\displaystyle n} equations, and is typically written in matrix form as y = X β ^ + e ^ {\displaystyle y=X{\hat {\beta }}+{\hat {e}}} , where y {\displaystyle y} and e ^ {\displaystyle {\hat {e}}} are n × 1 {\displaystyle n\times 1} vectors of dependent variable observations and residuals, respectively, X {\displaystyle X} is an n × k {\displaystyle n\times k} matrix of explanatory variables' observations, and β ^ {\displaystyle {\hat {\beta }}} is a coefficient vector. Then, the least squares solution is β ^ = ( X ′ X ) − 1 X ′ y {\displaystyle {\hat {\beta }}=(X^{\prime }X)^{-1}X^{\prime }y} yielding e ^ = y − X β ^ {\displaystyle {\hat {e}}=y-X{\hat {\beta }}} . In regressions estimated by least squares, it is common to refer to an explanatory variable's coefficient as the effect of that variable "holding constant" the other explanatory variables. For example, if wage is modeled as a function of education and work experience, the coefficient on education is interpreted as the difference in the expectation of wage for a unit difference in education, "holding constant" work experience. Econometrician Arthur Goldberger frames the Frisch–Waugh–Lovell theorem as "giving content to th[is] language".
Definition and interpretation The Frisch–Waugh–Lovell theorem states that in a least squares-estimated regression of the form
y i = β ^ 0 + β ^ 1 x 1 i + β ^ 2 x 2 i + ⋯ + β ^ k x k i + e ^ i {\displaystyle y_{i}={\hat {\beta }}_{0}+{\hat {\beta }}_{1}x_{1i}+{\hat {\beta }}_{2}x_{2i}+\cdots +{\hat {\beta }}_{k}x_{ki}+{\hat {e}}_{i}}
any coefficient β ^ j , j ∈ { 1 , … , k } {\displaystyle {\hat {\beta }}_{j},j\in \{1,\dots ,k\}} can be obtained by the two-step process of:
Regress x j {\displaystyle x_{j}} on the set of other explanatory variables, obtaining residuals x ~ j {\displaystyle {\tilde {x}}_{j}}
Regress y {\displaystyle y} on x ~ j {\displaystyle {\tilde {x}}_{j}} , obtaining β ^ j = cov ( y , x ~ j ) var ( x ~ j ) {\displaystyle {\hat {\beta }}_{j}={\frac {{\text{cov}}(y,{\tilde {x}}_{j})}{{\text{var}}({\tilde {x}}_{j})}}}
This two-step process is referred to as the residual regression or equivalently the regression anatomy theorem. This result is a numerical property of least squares estimation and does not depend on statistical properties of the data. The variation in each explanatory variable can be decomposed into two parts: the part which is linearly related to the set of other explanatory variables and the remaining, 'residual' variation. By the theorem, an explanatory variable's coefficient can be found from the simple regression of the dependent variable on the residual part. This result is the basis for interpreting the impact of including additional variables in a regression: it is equivalent to removing from the existing variables the component of shared variance with the new variables. Further, the Frisch–Waugh–Lovell theorem can be applied to understand how multicollinearity affects parameter estimation. When most of the variation in an explanatory variable is linearly explained by the other explanatory variables, the residual part has relatively very little variation. Consequently, the estimate of the explanatory variable's coefficient may be less precise than if fewer variables were controlled for.
Example Consider the regression of wage on education and parental income:
wage i = β ^ 0 + β ^ 1 education i + β ^ 2 parental income i + e ^ i {\displaystyle {\text{wage}}_{i}={\hat {\beta }}_{0}+{\hat {\beta }}_{1}{\text{education}}_{i}+{\hat {\beta }}_{2}{\text{parental income}}_{i}+{\hat {e}}_{i}}
While the least squares estimates for β ^ 0 , β ^ 1 {\displaystyle {\hat {\beta }}_{0},\,{\hat {\beta }}_{1}} and β ^ 2 {\displaystyle {\hat {\beta }}_{2}} can be obtained by minimizing ∑ i = 1 n e ^ i 2 {\textstyle \sum _{i=1}^{n}{\hat {e}}_{i}^{2}} directly, each can be equivalently obtained by the two-step residual regression. In the case of education:
Regress education on parental income, saving the residuals from this regression: the part of education not linearly related to parental income Regress wages on the residuals, obtaining the least squares estimate for β ^ 1 {\displaystyle {\hat {\beta }}_{1}}
This illustrates how β ^ 1 {\displaystyle {\hat {\beta }}_{1}} reflects the effect of education on wages controlling for parental income: it is the relationship between wages and the part of education not linearly related to parental income.
Double residual regression The double residual regression is the three-step process:
Regress x j {\displaystyle x_{j}} on the set of other explanatory variables, obtaining residuals x ~ j {\displaystyle {\tilde {x}}_{j}}
Regress y {\displaystyle y} on the set of explanatory variables excluding x j {\displaystyle x_{j}} , obtaining residuals y ~ {\displaystyle {\tilde {y}}}
Regress y ~ {\displaystyle {\tilde {y}}} on x ~ j {\displaystyle {\tilde {x}}_{j}} , estimating β ^ j = cov ( y ~ , x ~ j ) var ( x ~ j ) {\displaystyle {\hat {\beta }}_{j}={\frac {{\text{cov}}({\tilde {y}},{\tilde {x}}_{j})}{{\text{var}}({\tilde {x}}_{j})}}} and e ^ i = y ~ i − β ^ j x ~ j i {\displaystyle {\hat {e}}_{i}={\tilde {y}}_{i}-{\hat {\beta }}_{j}{\tilde {x}}_{ji}}
Like the two-step process, this yields an identical coefficient to the full regression. It includes the additional feature that the residuals from the regression in step 3 equal the residuals in the full regression.
Multivariate definition Consider the least squares-estimated regression y = X β ^ + Z δ ^ + e ^ {\displaystyle y=X{\hat {\beta }}+Z{\hat {\delta }}+{\hat {e}}} , where y {\displaystyle y} and e ^ {\displaystyle {\hat {e}}} are n × 1 {\displaystyle n\times 1} vectors of dependent variable observations and residuals, respectively, X {\displaystyle X} is an n × k {\displaystyle n\times k} matrix of k {\displaystyle k} explanatory variables' observations, Z {\displaystyle Z} is an n × p {\displaystyle n\times p} matrix of p {\displaystyle p} explanatory variables' observations, and β ^ {\displaystyle {\hat {\beta }}} and δ ^ {\displaystyle {\hat {\delta }}} are k × 1 {\displaystyle k\times 1} and p × 1 {\displaystyle p\times 1} coefficient vectors for X {\displaystyle X} and Z {\displaystyle Z} , respectively. Then, the Frisch–Waugh–Lovell theorem states that
β ^ = ( X ~ ′ X ~ ) − 1 X ~ ′ y = ( X ~ ′ X ~ ) − 1 X ~ ′ y ~ {\displaystyle {\hat {\beta }}=({\tilde {X}}^{\prime }{\tilde {X}})^{-1}{\tilde {X}}^{\prime }y=({\tilde {X}}^{\prime }{\tilde {X}})^{-1}{\tilde {X}}^{\prime }{\tilde {y}}}
e ^ = y ~ − X ~ β ^ {\displaystyle {\hat {e}}={\tilde {y}}-{\tilde {X}}{\hat {\beta }}}
where X ~ = X − Z ( Z ′ Z ) − 1 Z ′ X {\displaystyle {\tilde {X}}=X-Z(Z^{\prime }Z)^{-1}Z^{\prime }X} , the residuals from the least squares regression of X {\displaystyle X} on Z {\displaystyle Z} , and y ~ = y − Z ( Z ′ Z ) − 1 Z ′ y {\displaystyle {\tilde {y}}=y-Z(Z^{\prime }Z)^{-1}Z^{\prime }y} , the residuals from the least squares regression of y {\displaystyle y} on Z {\displaystyle Z} . The first expression of β ^ {\displaystyle {\hat {\beta }}} is the residual regression, while the second is the double residual regression.
Geometric interpretation With a linear regression of the form y = X β ^ + Z δ ^ + e ^ {\displaystyle y=X{\hat {\beta }}+Z{\hat {\delta }}+{\hat {e}}} , the fitted values X β ^ + Z δ ^ {\displaystyle X{\hat {\beta }}+Z{\hat {\delta }}} can be interpreted as the orthogonal projection of y {\displaystyle y} onto the column space of [ X Z ] {\displaystyle [X~Z]} , Col ( X , Z ) {\displaystyle {\text{Col}}(X,Z)} . The Frisch–Waugh–Lovell theorem is then (in the double residual regression case) the three step process:
Project X {\displaystyle X} onto the orthogonal complement of Col ( Z ) {\displaystyle {\text{Col}}(Z)} , obtaining residuals X ~ {\displaystyle {\tilde {X}}}
Project y {\displaystyle y} onto the orthogonal complement of Col ( Z ) {\displaystyle {\text{Col}}(Z)} , obtaining residual vector y ~ {\displaystyle {\tilde {y}}}
Project y ~ {\displaystyle {\tilde {y}}} onto Col ( X ~ ) {\displaystyle {\text{Col}}({\tilde {X}})} , obtaining projection X ~ β {\displaystyle {\tilde {X}}\beta } and residuals e = y ~ − X ~ β {\displaystyle e={\tilde {y}}-{\tilde {X}}\beta }
The resulting β ^ {\displaystyle {\hat {\beta }}} and residuals are identical to those in the full regression of y {\displaystyle y} on X {\displaystyle X} and Z {\displaystyle Z} .
Proof Consider the least squares-estimated regression y = X β ^ + Z δ ^ + e ^ {\displaystyle y=X{\hat {\beta }}+Z{\hat {\delta }}+{\hat {e}}} and annihilator matrix M Z = I − Z ( Z ′ Z ) − 1 Z ′ {\displaystyle M_{Z}=I-Z(Z^{\prime }Z)^{-1}Z^{\prime }} . Premultiplying both sides of the regression equation by the annihilator matrix removes from y {\displaystyle y} and X {\displaystyle X} the part linearly explained by Z {\displaystyle Z} :
M Z y = M Z ( X β ^ + Z δ ^ + e ^ ) y ~ = M Z X β ^ + M Z Z δ ^ + M Z e ^ y ~ = X ~ β ^ + e ^ {\displaystyle {\begin{aligned}M_{Z}y&=M_{Z}(X{\hat {\beta }}+Z{\hat {\delta }}+{\hat {e}})\\{\tilde {y}}&=M_{Z}X{\hat {\beta }}+M_{Z}Z{\hat {\delta }}+M_{Z}{\hat {e}}\\{\tilde {y}}&={\tilde {X}}{\hat {\beta }}+{\hat {e}}\end{aligned}}}
Where the third line follows from M Z Z = 0 {\displaystyle M_{Z}Z=0} by construction and M Z e ^ = e ^ {\displaystyle M_{Z}{\hat {e}}={\hat {e}}} from Z ′ e ^ = 0 {\displaystyle Z^{\prime }{\hat {e}}=0} , a property of least squares where e ^ {\displaystyle {\hat {e}}} is in the orthogonal complement of the column space of Z {\displaystyle Z} . Then, by the least squares solution, β ^ = ( X ~ ′ X ~ ) − 1 X ~ ′ y ~ {\displaystyle {\hat {\beta }}=({\tilde {X}}^{\prime }{\tilde {X}})^{-1}{\tilde {X}}^{\prime }{\tilde {y}}} and e ^ = y ~ − X ~ β ^ {\displaystyle {\hat {e}}={\tilde {y}}-{\tilde {X}}{\hat {\beta }}} . This concludes the proof of the double residual regression form of the Frisch-Waugh-Lovell theorem. The residual regression – where Z {\displaystyle Z} is projected out of X {\displaystyle X} but not y {\displaystyle y} – returns a coefficient vector equal to the double residual regression. Their equivalence can be proved:
( X ~ ′ X ~ ) − 1 X ~ ′ y = ( X ~ ′ X ~ ) − 1 X ′ M Z y = ( X ~ ′ X ~