A simple confusion
There are thousands of parameters in neural networks, and each one needs to be adjusted during training. The question is: How do we know which parameter, which direction, and how much to change when we only see the final output error value? Backpropagation is the solution to this problem, and it is also the key to the true success of deep learning.
Core idea: Chain rule
Backpropagation is essentially an efficient implementation of the chain rule in calculus. Consider the network as a nested series of functions: the input undergoes layers of transformations to obtain the output, which then calculates a loss. To ask what the derivative of the loss is with respect to a certain parameter, multiply back layer by layer along this calculation chain. Forward propagation calculates predictions and losses, while backpropagation returns error signals layer by layer from the output layer back to the input layer, calculating gradients for each layer along the way.
Why is it so fast
If we calculate the derivative of each parameter separately, the cost is outrageously high. The cleverness of backpropagation lies in reuse: the intermediate results (activation values, gradients of each layer) are only counted once and shared by multiple downstream parameters. This makes the cost of computing a complete gradient comparable to the magnitude of doing a forward propagation - it is this efficiency that makes large models with billions of parameters feasible in engineering.
What does a training cycle look like
- Forward propagation: Data flows through the network to obtain predicted values.
- Calculate loss: Use a loss function to measure the difference between predicted and true labels.
- Backpropagation: Starting from the loss, backpropagate layer by layer and calculate the gradient of each parameter.
- Parameter update: Optimizers (such as SGD, Adam) fine tune parameters according to the gradient direction.
- Repeat: Repeat the next batch of data until the loss converges.
Easy to step on pit
In deep networks, gradients need to be multiplied many times. If the derivative of each layer is too small, the gradient will decay all the way to near zero, which is called vanishing gradient; On the contrary, if it is too large, it will explode (explosive gradient). ReLU activation functions, residual connections (ResNet), batch normalization (BatchNorm), and gradient clipping are all practical methods designed to alleviate this problem.
Summary
Backpropagation is not some kind of magic, but an engineering implementation that organizes chain rules intelligently enough. It enables the network to attribute errors layer by layer and accurately allocate responsibility to each parameter - which is exactly the literal meaning of the phrase 'learn from errors' in deep learning.
【 Reference source 】 Comprehensive compilation of industry information publicly released (Ian Goodfellow et al., Deep Learning, MIT Press; Stanford CS231n course lecture notes; 3Blue1Brown neural network series).