What is regularization
In machine learning, overfitting is one of the most common and hidden problems: the model performs nearly perfectly on training data, and it is obviously inaccurate when encountering new data that has not been seen before. It did not truly 'learn the rules', but' memorized the answers'. Regularization is a method used to suppress rote memorization and enable models to learn more general patterns.
Why do models rely on rote memorization
When the model parameters are sufficient and the training data is limited, the model can completely reduce the training error to near zero by remembering the details of each sample. But in the real world, what is truly useful are often those smooth and stable trends, rather than the noise in the training samples. The core idea of regularization is to add an additional "penalty" to the training objective, constraining the model not to set parameters too extreme in order to accommodate individual samples.
Common practices
L2 regularization (weight decay)
Add a penalty term of the sum of squared parameters to the loss function to make the overall weight tend to be smaller. It corresponds to Ridge Regression in statistics, which can make decision boundaries smoother and is the most commonly used default method.
L1 regularization
Change the penalty term to the sum of the absolute values of the parameters, corresponding to Lasso. Its characteristic is that it directly compresses a portion of the weights to 0, thereby playing a role in feature selection - suitable for scenarios where key features need to be picked out at the same time.
Dropout
Randomly "turn off" a portion of neurons during training, forcing the network not to overly rely on a few nodes. This is equivalent to training a massive number of different sub networks simultaneously and integrating them during inference, which is a very popular regularization method in deep learning.
Early Stopping
Stop training promptly when the validation set error no longer decreases. It does not add additional parameters, but can effectively prevent the model from getting worse and worse on the training set. Data augmentation (flipping, cropping, etc.) is also an indirect form of regularization.
one-sentence summary
The essence of regularization is to 'trade a little training error for better generalization ability'. It's not about making the model learn less, but about making it learn more valuable parts. In practical work, L2 weight decay, Dropout, and early stop are often used together.
【 Reference source 】 Comprehensive compilation of publicly released machine learning textbooks and industry materials.