Why do models need to 'go down the mountain'
Training an AI model essentially involves finding a set of parameters that minimize prediction error. Imagine the error (loss) as the ups and downs of a mountain, where the parameter is your position and the goal is to reach the lowest point of the valley. Gradient Descent is the most commonly used "descent" strategy: with each step, first identify which direction is steepest (gradient) at the current position, and then take a small step towards the direction of the fastest descent.
Gradient: Tell you which side is going downhill
Gradient is a vector composed of the partial derivatives of the loss function for each parameter, pointing towards the direction where the function rises the fastest. So as long as you walk in the opposite direction of the gradient, the loss will decrease. The step size is controlled by the learning rate: if the step size is too large, it is easy to oscillate back and forth across the valley, while if it is too small, the descent will be too slow and training for a long time will not reach the bottom.
Three common gradient descent methods
- Batch Gradient Descent (BGD): Use all training data to calculate the gradient once and update it again. The direction is stable, but each time it is slow and memory is tight.
- Stochastic Gradient Descent (SGD): Using only one sample at a time, it updates quickly but with significant directional jitter, which actually helps to escape from local optima.
- Mini batch gradient descent: a compromise solution that uses a small batch (such as 32 or 128 samples) at a time, balancing speed and stability, is the de facto standard of deep learning today.
Why is actual training not just about 'following gradients'
The real loss surface is often not a smooth bowl, but covered with grooves and saddle points. As a result, improvements such as Momentum, AdaGrad, RMSProp, Adam, etc. have emerged: Momentum adds inertia to the ball descending the mountain, allowing it to pass through small pits; The adaptive method adjusts the step size separately for each parameter, allowing sparse updated parameters to progress steadily.
A few easy pitfalls to tread on
Setting the learning rate too high can result in fluctuating or even divergent losses; If it is too small, the convergence will be extremely slow. In practice, learning rate warm-up and decay (cosine, step) are commonly used to balance early stability and later refinement. In addition, when the gradient explosion occurs, gradient clipping is used to cover the bottom, and when the video memory is insufficient, gradient accumulation is used to make up an equivalent large batch.
Summary
Gradient descent is not mysterious: it is an iterative process of "looking at direction, taking small steps, and repeating". Understanding the source of gradients (backpropagation), step size (learning rate), and batch size is essential to grasping the main theme of model training.
【 Reference source 】 Comprehensive compilation of publicly released machine learning textbooks and industry technical materials.