Training a deep learning model from scratch often means massive amounts of data, expensive computing power, and lengthy time. But similar problems have actually been solved long ago: if there is already a model trained on a large dataset, can we borrow what it has learned and use it? Transfer Learning answers the question of transferring knowledge learned in one task to another related task.
1、 What is transfer learning
There are two core concepts in transfer learning: source domain and target domain. The source domain is a task that already has a large amount of annotated data, while the target domain is a task that we truly want to solve but with scarce data. The goal of transfer learning is to transfer the knowledge learned from the source domain (usually manifested as the weights of pre trained models) to the target domain, thereby achieving better results with less data and computing power.
2、 Why is it effective
Neural networks often learn general features at a shallow level, such as edges, textures, and colors in image tasks, and lexical and syntactic rules in natural language tasks; As we approach the deeper layers of output, the features become increasingly targeted towards specific tasks. Therefore, a pre trained model on a large dataset has universal low-level capabilities for many tasks, and only needs to adjust the high-level parts for new tasks. This is both intuitive and confirmed by numerous experiments.
3、 Two common practices
The first method is "feature extraction": treating the pre trained model as a fixed feature extractor, freezing all its weights, and only adding a new classification head at the end, training this small classification head with data from the new task. When data is scarce, this approach has the highest cost-effectiveness.
The second type is "fine-tuning": not only training the newly added layers, but also continuing to update some or even all of the weights of the pre trained model with a smaller learning rate, making the model adapt to the new task as a whole. When the data is relatively sufficient, fine-tuning can usually achieve better results.
4、 Transfer learning in the era of big models
In the era of big language models, transfer learning is almost the default paradigm: first pre train a general model with massive amounts of text, and then adapt it to specific scenarios through instruction fine-tuning, efficient parameter fine-tuning (such as LoRA), and even complete many tasks with just prompts. It can be said that the entire technical route of "pre training+fine-tuning" is the most successful practice of transfer learning ideas.
5、 Pits to be aware of when using
Transfer learning is not always effective. When there is a significant difference between the source domain and the target domain, the transferred knowledge may be counterproductive, a phenomenon known as negative transfer. In addition, the learning rate during fine-tuning should be significantly lower than that during pre training, otherwise valuable pre training knowledge may be 'washed away'; Freezing which layers and fine-tuning which layers often require experiments based on the amount of data.
6、 One sentence summary
Transfer learning transforms "reinventing the wheel" into "standing on the shoulders of giants": first lay a solid foundation with general knowledge, and then make fine adjustments for specific tasks. It is not only an engineering strategy that saves data and computing power, but also a key to understanding the sources of contemporary big model capabilities.
[Reference source]
Pan & amp; Yang，《A Survey on Transfer Learning》，IEEE Transactions on Knowledge and Data Engineering，2010； Yosinski et al, 《How transferable are features in deep neural networks?》，NeurIPS 2014。