In statistics, projection pursuit regression (PPR) is a statistical model developed by Jerome H. Friedman and Werner Stuetzle that extends additive models. This model adapts the additive models in that it first projects the data matrix of explanatory variables in the optimal direction before applying smoothing functions to these explanatory variables.
Model overview The model consists of linear combinations of ridge functions: non-linear transformations of linear combinations of the explanatory variables. The basic model takes the form
y i = β 0 + ∑ j = 1 r f j ( β j T x i ) + ε i , {\displaystyle y_{i}=\beta _{0}+\sum _{j=1}^{r}f_{j}(\beta _{j}^{\mathrm {T} }x_{i})+\varepsilon _{i},}
where xi is a 1 × p row of the design matrix containing the explanatory variables for example i, yi is a 1 × 1 prediction, {βj} is a collection of r vectors (each a unit vector of length p) which contain the unknown parameters, {fj} is a collection of r initially unknown smooth functions that map from R → R {\displaystyle \mathbb {R} \rightarrow \mathbb {R} } , and r is a hyperparameter. Good values for r can be determined through cross-validation or a forward stage-wise strategy which stops when the model fit cannot be significantly improved. As r approaches infinity and with an appropriate set of functions {fj}, the PPR model is a universal estimator, as it can approximate any continuous function in R p {\displaystyle \mathbb {R} ^{p}} .
Model estimation For a given set of data { ( y i , x i ) } i = 1 n {\displaystyle \{(y_{i},x_{i})\}_{i=1}^{n}} , the goal is to minimize the error function
S = ∑ i = 1 n [ y i − ∑ j = 1 r f j ( β j T x i ) ] 2 {\displaystyle S=\sum _{i=1}^{n}\left[y_{i}-\sum _{j=1}^{r}f_{j}(\beta _{j}^{\mathrm {T} }x_{i})\right]^{2}}
over the functions f j {\displaystyle f_{j}} and vectors β j {\displaystyle \beta _{j}} . No method exists for solving over all variables at once, but it can be solved via alternating optimization. First, consider each ( f j , β j ) {\displaystyle (f_{j},\beta _{j})} pair individually: Let all other parameters be fixed, and find a "residual", the variance of the output not accounted for by those other parameters, given by
r i = y i − ∑ l ≠ j f l ( β l T x i ) {\displaystyle r_{i}=y_{i}-\sum _{l\neq j}f_{l}(\beta _{l}^{\mathrm {T} }x_{i})}
The task of minimizing the error function now reduces to solving
… excerpt ends here. Continue reading the full article.
