When a big model completes a sentence, it faces tens of thousands or even tens of thousands of candidate words. Why does it pick out the next one? Why is the same question sometimes answered steadily and sometimes wildly? All of this is determined by the 'decoding strategy' - that is, how the model picks out that word from the probability distribution at each step. By understanding decoding, one can comprehend why the output of a large model is controllable and why it is difficult to fully control it.
1、 In fact, the model only gives a "probability table"
For each position generated by the big language model, the output is not a definite word, but a whole probability table (score for each candidate token in the word table). The decoding strategy is the rule that determines which word to choose based on this table. The selection method is different, and the same model can behave like two people.
2、 Greedy decoding
Simplicity: Choose the word with the highest probability for each step. The advantage is that it is deterministic and stable, and the same input always results in the same output; The disadvantage is that it is easy to make mistakes, and once a suboptimal word is selected in the early stages, it is difficult to pull it back, resulting in conservative and repetitive content.
3、 Beam Search
Not only focusing on the current best, but also retaining several candidate paths (beams) at the same time, expanding each path at each step and sorting them by cumulative probability, leaving the highest scoring ones, and finally selecting the one with the highest total probability of the entire sentence. It is more stable in tasks such as translation and summarization where the correct answer is relatively certain, but the computational workload increases exponentially and tends to be more general and secure in expression. Open ended creation can easily appear dull.
4、 Sampling: Give the model some randomness
1. Temperature: Adjust the "steepness" of the probability distribution. Low temperature is closer to greed and more stable; A higher temperature distribution is more evenly distributed and more divergent. Writing typically ranges from 0.7 to 1.0, while factual Q&A is even lower.
2. Top-k: Only sample from the top k words with the highest probability and cut off the long tail.
3. Top-p (kernel sampling): resample the candidate set up to p based on cumulative probability, which is more adaptive than Top-k - the set is smaller at high confidence and larger at low confidence.
5、 Repetitive punishment and other controls
Repetition penalty can lower the score of words that have already appeared, alleviating the phenomenon of "repeating machine"; In addition, there is constrained decoding, which forces the output to conform to formats such as JSON and regular, and is commonly used for structured output and tool calls.
6、 How to choose
To be stable and reproducible: greedy or low-temperature sampling. To be accurate and precise (translation/abstract): bundle search. To be creative and diverse: high temperature and Top-p. To make the program parsed: constraint decoding.
7、 One sentence summary
The ability of a large model is determined by its parameters, but its "personality" is largely determined by its decoding strategy. To make it obedient, lower the randomness; to make it flexible, let go of the randomness - mastering decoding is the fundamental skill for truly making good use of large models.
[Reference source] Comprehensive compilation of industry information that has been publicly released (such as Hugging Face's official documents on text generation and decoding strategies, relevant public papers, and mainstream inference framework documents).