The Hamilton-Jacobi-Bellman (HJB) equation is a nonlinear partial differential equation that provides necessary and sufficient conditions for optimality of a control with respect to a loss function. Its solution is the value function of the optimal control problem which, once known, can be used to obtain the optimal control by taking the maximizer (or minimizer) of the Hamiltonian involved in the HJB equation. The equation is a result of the theory of dynamic programming which was pioneered in the 1950s by Richard Bellman and coworkers. The connection to the Hamilton–Jacobi equation from classical physics was first drawn by Rudolf Kálmán. In discrete-time problems, the analogous difference equation is usually referred to as the Bellman equation. While classical variational problems, such as the brachistochrone problem, can be solved using the Hamilton–Jacobi–Bellman equation, the method can be applied to a broader spectrum of problems. Further it can be generalized to stochastic systems, in which case the HJB equation is a second-order elliptic partial differential equation. A major drawback, however, is that the HJB equation admits classical solutions only for a sufficiently smooth value function, which is not guaranteed in most situations. Instead, the notion of a viscosity solution is required, in which conventional derivatives are replaced by (set-valued) subderivatives.
Optimal Control Problems Consider the following problem in deterministic optimal control over the time period [ 0 , T ] {\displaystyle [0,T]} :
V ( x ( 0 ) , 0 ) = min u { ∫ 0 T C [ x ( t ) , u ( t ) ] d t + D [ x ( T ) ] } {\displaystyle V(x(0),0)=\min _{u}\left\{\int _{0}^{T}C[x(t),u(t)]\,dt+D[x(T)]\right\}}
where C [ ⋅ ] {\displaystyle C[\cdot ]} is the scalar cost rate function and D [ ⋅ ] {\displaystyle D[\cdot ]} is a function that gives the bequest value at the final state, x ( t ) {\displaystyle x(t)} is the system state vector, x ( 0 ) {\displaystyle x(0)} is assumed given, and u ( t ) {\displaystyle u(t)} for 0 ≤ t ≤ T {\displaystyle 0\leq t\leq T} is the control vector that we are trying to find. Thus, V ( x , t ) {\displaystyle V(x,t)} is the value function. The system must also be subject to
x ˙ ( t ) = F [ x ( t ) , u ( t ) ] {\displaystyle {\dot {x}}(t)=F[x(t),u(t)]\,}
where F [ ⋅ ] {\displaystyle F[\cdot ]} gives the vector determining physical evolution of the state vector over time.
The Partial Differential Equation For this simple system, the Hamilton–Jacobi–Bellman partial differential equation is
∂ V ( x , t ) ∂ t + min u { ∂ V ( x , t ) ∂ x ⋅ F ( x , u ) + C ( x , u ) } = 0 {\displaystyle {\frac {\partial V(x,t)}{\partial t}}+\min _{u}\left\{{\frac {\partial V(x,t)}{\partial x}}\cdot F(x,u)+C(x,u)\right\}=0}
subject to the terminal condition
V ( x , T ) = D ( x ) , {\displaystyle V(x,T)=D(x),\,}
As before, the unknown scalar function V ( x , t ) {\displaystyle V(x,t)} in the above partial differential equation is the Bellman value function, which represents the cost incurred from starting in state x {\displaystyle x} at time t {\displaystyle t} and controlling the system optimally from then until time T {\displaystyle T} .
Deriving the Equation Intuitively, the HJB equation can be derived as follows. If V ( x ( t ) , t ) {\displaystyle V(x(t),t)} is the optimal cost-to-go function (also called the 'value function'), then by Richard Bellman's principle of optimality, going from time t to t + dt, we have
… excerpt ends here. Continue reading the full article.
