Time delay neural network (TDNN) is a multilayer artificial neural network architecture whose purpose is to 1) classify patterns with shift-invariance, and 2) model context at each layer of the network. It is essentially a 1-d convolutional neural network (CNN). Shift-invariant classification means that the classifier does not require explicit segmentation prior to classification. For the classification of a temporal pattern (such as speech), the TDNN thus avoids having to determine the beginning and end points of sounds before classifying them. For contextual modelling in a TDNN, each neural unit at each layer receives input not only from activations/features at the layer below, but from a pattern of unit output and its context. For time signals each unit receives as input the activation patterns over time from units below. Applied to two-dimensional classification (images, time-frequency patterns), the TDNN can be trained with shift-invariance in the coordinate space and avoids precise segmentation in the coordinate space.
History The TDNN was introduced in the late 1980s and applied to a task of phoneme classification for automatic speech recognition in speech signals where the automatic determination of precise segments or feature boundaries was difficult or impossible. Because the TDNN recognizes phonemes and their underlying acoustic/phonetic features, independent of position in time, it improved performance over static classification. It was also applied to two-dimensional signals (time-frequency patterns in speech, and coordinate space pattern in OCR). Kunihiko Fukushima published the neocognitron in 1980. Max pooling appears in a 1982 publication on the neocognitron and was in the 1989 publication in LeNet-5. In 1990, Yamaguchi et al. used max pooling in TDNNs in order to realize a speaker independent isolated word recognition system.
Overview
Architecture In modern language, the design of TDNN is a 1D convolutional neural network, where the direction of convolution is across the dimension of time. In the original design, there are exactly 3 layers. The input to the network is a continuous speech signal, preprocessed into a 2D array (a mel scale spectrogram). One dimension is time at 10 ms per frame, and the other dimension is frequency. The time dimension can be arbitrarily long, but the frequency dimension was only 16-long. In the original experiment, they only considered very short speech signals pronouncing single words like "baa", "daa", "gaa". Because of this, the speech signals could be very short, indeed, only 15 frames long (150 ms in time). In detail, they processed a voice signal as follows:
Input speech is sampled at 12 kHz, Hamming-windowed. Its FFT is computed every 5 ms. The mel scale coefficients are computed from the power spectrum by taking log energies in each mel scale energy band. Adjacent coefficients in time are soothed over, resulting in one frame every 10 ms. For each signal, a human manually detect the onset of the vowel, and the entire speech signal is cut off except 7 frames before and 7 frames after, leaving just 15 frames in total, centered at the onset of the vowel. The coefficients are normalized by subtracting the mean, then scaling, so that the signals fall between -1 and +1. The first layer of the TDNN is a 1D convolutional layer. The layer contains 8 kernels of shape 3 × 16 {\displaystyle 3\times 16} . It outputs a tensor of shape 8 × 13 {\displaystyle 8\times 13} . The second layer of the TDNN is a 1D convolutional layer. The layer contains 3 kernels of shape 5 × 8 {\displaystyle 5\times 8} . It outputs a tensor of shape 3 × 9 {\displaystyle 3\times 9} . The third layer of the TDNN is not a convolutional layer. Instead, it is simply a fixed layer with 3 neurons. Let the output from the second layer be x i , j {\displaystyle x_{i,j}} where i ∈ 1 : 3 {\displaystyle i\in 1:3} and j ∈ 1 : 9 {\displaystyle j\in 1:9} . The i {\displaystyle i} -th neuron in the third layer computes σ ( ∑ j ∈ 1 : 9 x i , j ) {\displaystyle \sigma (\sum _{j\in 1:9}x_{i,j})} , where σ {\displaystyle \sigma } is the sigmoid function. Essentially, it can be thought of as a convolution layer with 3 kernels of shape 1 × 9 {\displaystyle 1\times 9} . It was trained on ~800 samples for 20000–50000 backpropagation steps. Each steps was computed in a batch over the entire training dataset, i.e. not stochastic. It required the use of an Alliant supercomputer with 4 processors.
… excerpt ends here. Continue reading the full article.
