In statistics, the projection matrix ( P ) {\displaystyle (\mathbf {P} )} , sometimes also called the influence matrix or hat matrix ( H ) {\displaystyle (\mathbf {H} )} , maps the vector of response values (dependent variable values) to the vector of fitted values (or predicted values). It describes the influence each response value has on each fitted value. The diagonal elements of the projection matrix are the leverages, which describe the influence each response value has on the fitted value for that same observation.
Definition If the vector of response values is denoted by y {\displaystyle \mathbf {y} } and the vector of fitted values by y ^ {\displaystyle \mathbf {\hat {y}} } ,
y ^ = P y . {\displaystyle \mathbf {\hat {y}} =\mathbf {P} \mathbf {y} .}
As y ^ {\displaystyle \mathbf {\hat {y}} } is usually pronounced "y-hat", the projection matrix P {\displaystyle \mathbf {P} } is also named hat matrix as it "puts a hat on y {\displaystyle \mathbf {y} } ".
Application for residuals The formula for the vector of residuals r {\displaystyle \mathbf {r} } can also be expressed compactly using the projection matrix:
r = y − y ^ = y − P y = ( I − P ) y . {\displaystyle \mathbf {r} =\mathbf {y} -\mathbf {\hat {y}} =\mathbf {y} -\mathbf {P} \mathbf {y} =\left(\mathbf {I} -\mathbf {P} \right)\mathbf {y} .}
where I {\displaystyle \mathbf {I} } is the identity matrix. The matrix M := I − P {\displaystyle \mathbf {M} :=\mathbf {I} -\mathbf {P} } is sometimes referred to as the residual maker matrix or the annihilator matrix. The covariance matrix of the residuals r {\displaystyle \mathbf {r} } , by error propagation, equals
Σ r = ( I − P ) T Σ ( I − P ) {\displaystyle \mathbf {\Sigma } _{\mathbf {r} }=\left(\mathbf {I} -\mathbf {P} \right)^{\textsf {T}}\mathbf {\Sigma } \left(\mathbf {I} -\mathbf {P} \right)} , where Σ {\displaystyle \mathbf {\Sigma } } is the covariance matrix of the error vector (and by extension, the response vector as well). For the case of linear models with independent and identically distributed errors in which Σ = σ 2 I {\displaystyle \mathbf {\Sigma } =\sigma ^{2}\mathbf {I} } , this reduces to:
Σ r = ( I − P ) σ 2 {\displaystyle \mathbf {\Sigma } _{\mathbf {r} }=\left(\mathbf {I} -\mathbf {P} \right)\sigma ^{2}} .
Intuition
From the figure, it is clear that the closest point from the vector b {\displaystyle \mathbf {b} } onto the column space of A {\displaystyle \mathbf {A} } , is A x {\displaystyle \mathbf {Ax} } , and is one where we can draw a line orthogonal to the column space of A {\displaystyle \mathbf {A} } . A vector that is orthogonal to the column space of a matrix is in the nullspace of the matrix transpose, so
A T ( b − A x ) = 0 {\displaystyle \mathbf {A} ^{\textsf {T}}(\mathbf {b} -\mathbf {Ax} )=0} . From there, one rearranges, so
… excerpt ends here. Continue reading the full article.
