In functional analysis, a reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions in which point evaluation is a continuous linear functional. Specifically, a Hilbert space H {\displaystyle H} of functions from a set X {\displaystyle X} (to R {\displaystyle \mathbb {R} } or C {\displaystyle \mathbb {C} } ) is an RKHS if the point-evaluation functional L x : H → C {\displaystyle L_{x}:H\to \mathbb {C} } , L x ( f ) = f ( x ) {\displaystyle L_{x}(f)=f(x)} , is continuous for every x ∈ X {\displaystyle x\in X} . Equivalently, H {\displaystyle H} is an RKHS if there exists a function K x ∈ H {\displaystyle K_{x}\in H} such that, for all f ∈ H {\displaystyle f\in H} , ⟨ f , K x ⟩ = f ( x ) . {\displaystyle \langle f,K_{x}\rangle =f(x).} The function K x {\displaystyle K_{x}} is then called the reproducing kernel, and it reproduces the value of f {\displaystyle f} at x {\displaystyle x} via the inner product. An immediate consequence of this property is that convergence in norm implies uniform convergence on any subset of X {\displaystyle X} on which ‖ K x ‖ {\displaystyle \|K_{x}\|} is bounded. However, the converse does not necessarily hold. Often the set X {\displaystyle X} carries a topology, and ‖ K x ‖ {\displaystyle \|K_{x}\|} depends continuously on x ∈ X {\displaystyle x\in X} , in which case: convergence in norm implies uniform convergence on compact subsets of X {\displaystyle X} . It is not entirely straightforward to construct natural examples of a Hilbert space which are not an RKHS in a non-trivial fashion. Some examples, however, have been found. While, formally, L2 spaces are defined as Hilbert spaces of equivalence classes of functions, this definition can trivially be extended to a Hilbert space of functions by choosing a (total) function as a representative for each equivalence class. However, no choice of representatives can make this space an RKHS ( K 0 {\displaystyle K_{0}} would need to be the non-existent Dirac delta function). However, there are RKHSs in which the norm is an L2-norm, such as the space of band-limited functions (see the example below). An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every x {\displaystyle x} in the set on which the functions are defined, "evaluation at x {\displaystyle x} " can be performed by taking an inner product with a function determined by the kernel. Such a reproducing kernel exists if and only if every evaluation functional is continuous. The reproducing kernel was first introduced in the 1907 work of Stanisław Zaremba concerning boundary value problems for harmonic and biharmonic functions. James Mercer simultaneously examined functions which satisfy the reproducing property in the theory of integral equations. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of Gábor Szegő, Stefan Bergman, and Salomon Bochner. The subject was eventually systematically developed in the early 1950s by Nachman Aronszajn and Stefan Bergman. These spaces have wide applications, including complex analysis, harmonic analysis, and quantum mechanics. Reproducing kernel Hilbert spaces are particularly important in the field of statistical learning theory because of the celebrated representer theorem which states that every function in an RKHS that minimises an empirical risk functional can be written as a linear combination of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the empirical risk minimization problem from an infinite dimensional to a finite dimensional optimization problem. For ease of understanding, we provide the framework for real-valued Hilbert spaces. The theory can be easily extended to spaces of complex-valued functions and hence include the many important examples of reproducing kernel Hilbert spaces that are spaces of analytic functions.
Definition Let X {\displaystyle X} be an arbitrary set and H {\displaystyle H} a Hilbert space of real-valued functions on X {\displaystyle X} , equipped with pointwise addition and pointwise scalar multiplication. The evaluation functional over the Hilbert space of functions H {\displaystyle H} is a linear functional that evaluates each function at a point x {\displaystyle x} ,
… excerpt ends here. Continue reading the full article.
