Powell's dog leg method, also called Powell's hybrid method, is an iterative optimisation algorithm for the solution of non-linear least squares problems, introduced in 1970 by Michael J. D. Powell. Similarly to the Levenberg–Marquardt algorithm, it combines the Gauss–Newton algorithm with gradient descent, but it uses an explicit trust region. At each iteration, if the step from the Gauss–Newton algorithm is within the trust region, it is used to update the current solution. If not, the algorithm searches for the minimum of the objective function along the steepest descent direction, known as Cauchy point. If the Cauchy point is outside of the trust region, it is truncated to the boundary of the latter and it is taken as the new solution. If the Cauchy point is inside the trust region, the new solution is taken at the intersection between the trust region boundary and the line joining the Cauchy point and the Gauss-Newton step (dog leg step). The name of the method derives from the resemblance between the construction of the dog leg step and the shape of a dogleg hole in golf.
Formulation
Given a least squares problem in the form
F ( x ) = 1 2 ‖ f ( x ) ‖ 2 = 1 2 ∑ i = 1 m ( f i ( x ) ) 2 {\displaystyle F({\boldsymbol {x}})={\frac {1}{2}}\left\|{\boldsymbol {f}}({\boldsymbol {x}})\right\|^{2}={\frac {1}{2}}\sum _{i=1}^{m}\left(f_{i}({\boldsymbol {x}})\right)^{2}}
with f i : R n → R {\displaystyle f_{i}:\mathbb {R} ^{n}\to \mathbb {R} } , Powell's dog leg method finds the optimal point x ∗ = argmin x F ( x ) {\displaystyle {\boldsymbol {x}}^{*}=\operatorname {argmin} _{\boldsymbol {x}}F({\boldsymbol {x}})} by constructing a sequence x k = x k − 1 + δ k {\displaystyle {\boldsymbol {x}}_{k}={\boldsymbol {x}}_{k-1}+\delta _{k}} that converges to x ∗ {\displaystyle {\boldsymbol {x}}^{*}} . At a given iteration, the Gauss–Newton step is given by
δ g n = − ( J ⊤ J ) − 1 J ⊤ f ( x ) {\displaystyle {\boldsymbol {\delta _{gn}}}=-\left({\boldsymbol {J}}^{\top }{\boldsymbol {J}}\right)^{-1}{\boldsymbol {J}}^{\top }{\boldsymbol {f}}({\boldsymbol {x}})}
where J = ( ∂ f i ∂ x j ) {\displaystyle {\boldsymbol {J}}=\left({\frac {\partial {f_{i}}}{\partial {x_{j}}}}\right)} is the Jacobian matrix, while the steepest descent direction is given by
δ s d = − J ⊤ f ( x ) . {\displaystyle {\boldsymbol {\delta _{sd}}}=-{\boldsymbol {J}}^{\top }{\boldsymbol {f}}({\boldsymbol {x}}).}
The objective function is linearised along the steepest descent direction
… excerpt ends here. Continue reading the full article.
