Kernel methods are a well-established tool to analyze the relationship between input data and the corresponding output of a function. Kernels encapsulate the properties of functions in a computationally efficient way and allow algorithms to easily swap functions of varying complexity. In typical machine learning algorithms, these functions produce a scalar output. Recent development of kernel methods for functions with vector-valued output is due, at least in part, to interest in simultaneously solving related problems. Kernels which capture the relationship between the problems allow them to borrow strength from each other. Algorithms of this type include multi-task learning (also called multi-output learning or vector-valued learning), transfer learning, and co-kriging. Multi-label classification can be interpreted as mapping inputs to (binary) coding vectors with length equal to the number of classes. In Gaussian processes, kernels are called covariance functions. Multiple-output functions correspond to considering multiple processes. See Bayesian interpretation of regularization for the connection between the two perspectives.
History The history of learning vector-valued functions is closely linked to transfer learning- storing knowledge gained while solving one problem and applying it to a different but related problem. The fundamental motivation for transfer learning in the field of machine learning was discussed in a NIPS-95 workshop on “Learning to Learn”, which focused on the need for lifelong machine learning methods that retain and reuse previously learned knowledge. Research on transfer learning has attracted much attention since 1995 in different names: learning to learn, lifelong learning, knowledge transfer, inductive transfer, multitask learning, knowledge consolidation, context-sensitive learning, knowledge-based inductive bias, metalearning, and incremental/cumulative learning. Interest in learning vector-valued functions was particularly sparked by multitask learning, a framework which tries to learn multiple, possibly different tasks simultaneously. Much of the initial research in multitask learning in the machine learning community was algorithmic in nature, and applied to methods such as neural networks, decision trees and k-nearest neighbors in the 1990s. The use of probabilistic models and Gaussian processes was pioneered and largely developed in the context of geostatistics, where prediction over vector-valued output data is known as cokriging. Geostatistical approaches to multivariate modeling are mostly formulated around the linear model of coregionalization (LMC), a generative approach for developing valid covariance functions that has been used for multivariate regression and in statistics for computer emulation of expensive multivariate computer codes. The regularization and kernel theory literature for vector-valued functions followed in the 2000s. While the Bayesian and regularization perspectives were developed independently, they are in fact closely related.
Notation In this context, the supervised learning problem is to learn the function f {\displaystyle f} which best predicts vector-valued outputs y i {\displaystyle \mathbf {y_{i}} } given inputs (data) x i {\displaystyle \mathbf {x_{i}} } .
f ( x i ) = y i {\displaystyle f(\mathbf {x_{i}} )=\mathbf {y_{i}} } for i = 1 , … , N {\displaystyle i=1,\ldots ,N}
x i ∈ X {\displaystyle \mathbf {x_{i}} \in {\mathcal {X}}} , an input space (e.g. X = R p {\displaystyle {\mathcal {X}}=\mathbb {R} ^{p}} )
y i ∈ R D {\displaystyle \mathbf {y_{i}} \in \mathbb {R} ^{D}}
In general, each component of ( y i {\displaystyle \mathbf {y_{i}} } ), could have different input data ( x d , i {\displaystyle \mathbf {x_{d,i}} } ) with different cardinality ( p {\displaystyle p} ) and even different input spaces ( X {\displaystyle {\mathcal {X}}} ). Geostatistics literature calls this case heterotopic, and uses isotopic to indicate that the each component of the output vector has the same set of inputs. Here, for simplicity in the notation, we assume the number and sample space of the data for each output are the same.
Regularization perspective Sources: From the regularization perspective, the problem is to learn f ∗ {\displaystyle f_{*}} belonging to a reproducing kernel Hilbert space of vector-valued functions ( H {\displaystyle {\mathcal {H}}} ). This is similar to the scalar case of Tikhonov regularization, with some extra care in the notation.
… excerpt ends here. Continue reading the full article.
