In machine learning, normalization is a statistical technique with various applications. There are two main forms of normalization, namely data normalization and activation normalization. Data normalization (or feature scaling) includes methods that rescale input data so that the features have the same range, mean, variance, or other statistical properties. For instance, a popular choice of feature scaling method is min-max normalization, where each feature is transformed to have the same range (typically [ 0 , 1 ] {\displaystyle [0,1]} or [ − 1 , 1 ] {\displaystyle [-1,1]} ). This solves the problem of different features having vastly different scales, for example if one feature is measured in kilometers and another in nanometers. Activation normalization, on the other hand, is specific to deep learning, and includes methods that rescale the activation of hidden neurons inside neural networks. Normalization is often used to:
increase the speed of training convergence, reduce sensitivity to variations and feature scales in input data, reduce overfitting, and produce better model generalization to unseen data. Normalization techniques are often theoretically justified as reducing covariance shift, smoothing optimization landscapes, and increasing regularization, though they are mainly justified by empirical success.
Batch normalization Batch normalization (BatchNorm) operates on the activations of a layer for each mini-batch. Consider a simple feedforward network, defined by chaining together modules:
x ( 0 ) ↦ x ( 1 ) ↦ x ( 2 ) ↦ ⋯ {\displaystyle x^{(0)}\mapsto x^{(1)}\mapsto x^{(2)}\mapsto \cdots }
where each network module can be a linear transform, a nonlinear activation function, a convolution, etc. x ( 0 ) {\displaystyle x^{(0)}} is the input vector, x ( 1 ) {\displaystyle x^{(1)}} is the output vector from the first module, etc. BatchNorm is a module that can be inserted at any point in the feedforward network. For example, suppose it is inserted just after x ( l ) {\displaystyle x^{(l)}} , then the network would operate accordingly:
⋯ ↦ x ( l ) ↦ B N ( x ( l ) ) ↦ x ( l + 1 ) ↦ ⋯ {\displaystyle \cdots \mapsto x^{(l)}\mapsto \mathrm {BN} (x^{(l)})\mapsto x^{(l+1)}\mapsto \cdots }
The BatchNorm module does not operate over individual inputs. Instead, it must operate over one batch of inputs at a time. Concretely, suppose we have a batch of inputs x ( 1 ) ( 0 ) , x ( 2 ) ( 0 ) , … , x ( B ) ( 0 ) {\displaystyle x_{(1)}^{(0)},x_{(2)}^{(0)},\dots ,x_{(B)}^{(0)}} , fed all at once into the network. We would obtain in the middle of the network some vectors:
x ( 1 ) ( l ) , x ( 2 ) ( l ) , … , x ( B ) ( l ) {\displaystyle x_{(1)}^{(l)},x_{(2)}^{(l)},\dots ,x_{(B)}^{(l)}}
The BatchNorm module computes the coordinate-wise mean and variance of these vectors:
… excerpt ends here. Continue reading the full article.
