When training large models we often hit a conflict: we want a larger batch size for more stable gradient estimates, but GPU memory will not fit it. Gradient accumulation is the classic trick that resolves this by "saving up" several small batches into one equivalent large batch.
1. Why large batches
Batch size affects training in two ways. First, stability of the gradient estimate: the larger the batch, the closer the estimated gradient is to the true gradient over all data, giving smoother updates. Second, hardware utilization: a moderate batch keeps the GPU parallel units busy and improves throughput.
But a larger batch consumes more memory. Backpropagation must keep the intermediate results (activations) of every layer from the forward pass, so doubling the batch often nearly doubles activation memory. When the model itself is large, the batch is often pushed very small, sometimes down to one.
2. The core idea
Gradient accumulation is straightforward: run forward and backward on a small batch, but do not update the parameters yet; instead accumulate the gradients. After repeating this for several small batches, update the parameters once using the accumulated gradient, then clear it.
For example, if you want an effective batch of 32 but memory only allows 8, you can process four batches of 8. The first three only accumulate gradients, and only the fourth triggers a parameter update. The update then draws on 4 x 8 = 32 samples, close to using batch 32 directly, while memory only ever holds a batch of 8.
3. Two easily missed details
First, the loss must be scaled by the number of accumulation steps. If you simply add the gradients of each small batch, the accumulated magnitude becomes several times that of a single batch, silently scaling up the learning rate. The usual fix is to divide each small batch loss by the number of accumulation steps, or to average the gradient after accumulation.
Second, operations that depend on batch statistics differ. When the model uses batch normalization, the statistics from accumulated small batches are not identical to those of a truly large batch. Modern large language models mostly use LayerNorm or RMSNorm, which do not depend on the batch dimension, so this effect is much smaller; random operations such as Dropout still run independently per small batch and add subtle differences.
4. When to use it, when not to
Gradient accumulation suits three cases: limited single-GPU memory but a desire for the stability of larger batches; multi-GPU training where you want a larger global batch without more communication pressure; and reproducing a paper that uses a large batch when local hardware cannot hold it directly.
The cost is slower training: the same number of samples needs more forward/backward passes per update, and with fewer updates and limited parallelism, wall-clock time usually grows. So gradient accumulation is a classic trade-time-for-memory choice, in the same family as gradient checkpointing and mixed precision.
Summary
Gradient accumulation changes neither the model structure nor the parameters; it only changes how often the parameters are updated. Understanding it makes clear that it does not really enlarge the batch on the hardware, but makes the optimizer feel it saw a larger batch. The key to using it well is to scale the loss correctly and to be aware of the subtle differences from batch-dependent operations such as batch normalization.
Reference: compiled from publicly released industry information.