The larger the model, the higher the inference cost. There are three common cost reduction methods in the industry: quantifying pressure accuracy, distillation to replace small models, and one often overlooked method - pruning. This article explains in plain language what pruning is, how to prune it, and the difference between it and the other two paths.
1、 What is model pruning
The core idea of pruning is simple: not every parameter in a neural network is equally important. Pruning is the process of identifying weights, neurons, or attention heads that contribute very little, removing them, and making the model sparser and lighter.
This idea is not new. In 1989, LeCun et al. proposed Optimal Brain Damage, which uses second-order information to determine which weights can be safely removed. Since then, pruning has been a classic direction for model compression.
2、 Three common cutting methods
-Unstructured pruning: deleting weights one by one, with low accuracy loss, but the result is a sparse matrix, which may not be accelerated by ordinary hardware.
-Structured pruning: cutting channels, attention heads, or layers in whole blocks, with regular shapes, often leads to faster inference.
-Semi structured pruning: a compromise solution, such as N: M sparsity (retaining N out of every M weights), which has both sparsity and can be utilized by the sparse computing units of modern GPUs.
3、 How to do pruning: explain the process clearly in one go
1. Training or loading: First, there is a trained dense model.
2. Evaluate importance: Rate each unit based on the absolute weight, gradient, or activation statistic.
3. Cut off low scoring units: proportionally remove the parts with the lowest scores.
4. Fine tuning recovery: After cutting, there is usually a decrease in accuracy. Train the model again with a portion of the data to "recover" its state.
5. Iteration: Some methods will "cut a little, train a little" repeatedly, which is more stable than cutting too hard at once.
4、 The difference between quantification and distillation
-Quantification: Do not delete parameters, but reduce the numerical accuracy of each parameter (such as FP16 → INT8/INT4), saving memory and bandwidth.
-Distillation: Train a small "student model" to mimic the behavior of a larger model, replacing it with the model structure.
-Pruning: Directly deleting some existing structures, the overall architecture remains unchanged, only sparser.
The three are not mutually exclusive - in actual deployment, they are often used in combination of "pruning first, then quantifying".
5、 Precautions in practice
-The more severe the cutting, the more obvious the loss of accuracy, which needs to be compensated by fine-tuning; A one size fits all approach to extremely high sparsity is almost certain to collapse.
-Unstructured pruning may not necessarily be faster on general hardware, and for true acceleration, structured or semi-structured pruning should be prioritized.
-Whether it's worth cutting depends on whether your bottleneck is video memory, latency, or throughput - different methods apply to different goals.
6、 Summary
Pruning is the representative of "subtraction" in model compression: it is not about changing precision or architecture, but directly deleting redundant parts. It, together with quantification and distillation, constitutes the three mainstream paths for cost reduction in current large-scale models.
【 Reference Source 】 Comprehensive compilation of industry information released publicly