Why do we need attention mechanisms
Not every word is equally important when dealing with a sentence or text. Traditional recurrent neural networks (RNNs) read in sequence one by one, relying on a "memory" vector to compress historical information together, making it easy to forget the previous content once the sentence is long. The attention mechanism has changed its approach: allowing the model to "look back" at the entire input at each step and dynamically determine which parts to focus on.
Query, Key, Value: An Intuitive Metaphor
Attention is often divided into three roles: Query, Key, and Value. You can think of it as a search: you compare a bunch of tags (keys) with a question (Query), and take more information (Value) from whoever's tag matches the question more. The matching degree is calculated using similarity, and then normalized to obtain the "attention weight".
Self attention: Let each word read the entire sentence
When the query, key, and value all come from the same input, self attention is obtained. Taking the sentence "The little cat is running after the ball because it is very happy" as an example, the model can allocate more attention to the "little cat" instead of the "ball" through attention weights when understanding "it", thus correctly judging the referential relationship. This is precisely the key to its ability to capture long-range dependencies.
Multi-head attention
With only one set of attention, models often only focus on one type of relationship. Multi Head Attention runs multiple sets of Query/Key/Value in parallel, allowing different "heads" to focus on different levels of patterns such as syntax, reference, and position, and finally concatenate them together. The reason why Transformer can become the backbone of large language models is due to the attention mechanism.
one-sentence summary
The attention mechanism solves the problem of "where to look": it allows the model to no longer treat all inputs equally, but dynamically allocate attention based on the current task. From machine translation to large-scale model dialogue, it is one of the most essential components.
【 Reference source 】 Comprehensive compilation of publicly released machine learning textbooks and industry materials.