Mixture of experts (MoE) is a machine learning technique where multiple expert networks (learners) are used to divide a problem space into homogeneous regions. MoE represents a form of ensemble learning. They were also called committee machines.
Basic theory MoE always has the following components, but they are implemented and combined differently according to the problem being solved:
Experts f 1 , . . . , f n {\displaystyle f_{1},...,f_{n}} , each taking the same input x {\displaystyle x} , and producing outputs f 1 ( x ) , . . . , f n ( x ) {\displaystyle f_{1}(x),...,f_{n}(x)} . A weighting function (also known as a gating function) w {\displaystyle w} , which takes input x {\displaystyle x} and produces a vector of outputs ( w ( x ) 1 , . . . , w ( x ) n ) {\displaystyle (w(x)_{1},...,w(x)_{n})} . This may or may not be a probability distribution, but in both cases, its entries are non-negative.
θ = ( θ 0 , θ 1 , . . . , θ n ) {\displaystyle \theta =(\theta _{0},\theta _{1},...,\theta _{n})} is the set of parameters. The parameter θ 0 {\displaystyle \theta _{0}} is for the weighting function. The parameters θ 1 , … , θ n {\displaystyle \theta _{1},\dots ,\theta _{n}} are for the experts. Given an input x {\displaystyle x} , the mixture of experts produces a single output by combining f 1 ( x ) , . . . , f n ( x ) {\displaystyle f_{1}(x),...,f_{n}(x)} according to the weights w ( x ) 1 , . . . , w ( x ) n {\displaystyle w(x)_{1},...,w(x)_{n}} in some way, usually by f ( x ) = ∑ i w ( x ) i f i ( x ) {\displaystyle f(x)=\sum _{i}w(x)_{i}f_{i}(x)} . Both the experts and the weighting function are trained by minimizing some loss function, generally via gradient descent. There is much freedom in choosing the precise form of experts, the weighting function, and the loss function.
Meta-pi network The meta-pi network, reported by Hampshire and Waibel, uses f ( x ) = ∑ i w ( x ) i f i ( x ) {\displaystyle f(x)=\sum _{i}w(x)_{i}f_{i}(x)} as the output. The model is trained by performing gradient descent on the mean-squared error loss L := 1 N ∑ k ‖ y k − f ( x k ) ‖ 2 {\displaystyle L:={\frac {1}{N}}\sum _{k}\|y_{k}-f(x_{k})\|^{2}} . The experts may be arbitrary functions. In their original publication, they were solving the problem of classifying phonemes in speech signal from 6 different Japanese speakers, 2 females and 4 males. They trained 6 experts, each being a "time-delayed neural network" (essentially a multilayered convolution network over the mel spectrogram). They found that the resulting mixture of experts dedicated 5 experts for 5 of the speakers, but the 6th (male) speaker does not have a dedicated expert, instead his voice was classified by a linear combination of the experts for the other 3 male speakers.
… excerpt ends here. Continue reading the full article.
