A pre trained large model is essentially a machine that predicts the next word. It can continue writing text, but may not necessarily be willing to listen - if you ask it a question, it may continue to weave stories on its own. Turning such primitive models into assistants that can chat, follow rules, and avoid harmful content relies on an additional process: alignment, and RLHF is one of the most widely influential technological routes.
1、 From 'being able to speak' to 'speaking well'
Pre training addresses' language proficiency ', but does not address' intent alignment'. The goal of the model is to maximize the statistical probability of generating text, rather than satisfying humans. It can write smooth paragraphs, but it doesn't know which content to reject and which answers better meet people's expectations. Therefore, it is necessary to 'teach' human preferences to the model.
2、 The Three Steps of RLHF
Step 1, Supervised Fine tuning (SFT): Use high-quality question and answer examples written manually to teach the model the basic format and style of "what is a good answer". The second step is to train a reward model: have the annotator rank multiple answers to the same question, and use these preference data to train a model that can score the answers, which is equivalent to a "proxy judge of human preferences". The third step is reinforcement learning: using a reward model as a referee, encouraging the model to produce higher scoring answers through strategy optimization (common algorithms such as PPO), while constraining it not to deviate too far from its original abilities.
3、 Alignment should simultaneously consider three things
A mature alignment scheme typically needs to balance three points: helpful (truly solving the problem), honest (not fabricating or acknowledging uncertainty), and harmless (rejecting dangerous requests). There is often tension between the three - excessive pursuit of harmlessness will make the model perfunctory; Excessive pursuit of usefulness may also relax safety boundaries. How to make choices is not only an engineering issue, but also a matter of values.
4、 Controversy and Evolution
RLHF relies heavily on manual annotation, which is costly and time-consuming, and the "human preference" itself carries subjectivity and annotator bias. Therefore, the industry is also exploring more labor-saving or stable supplementary solutions, such as using AI to score answers (RLAIF), rule-based rewards, and lighter methods such as direct preference optimization (DPO). Alignment is not a one-time project, but a continuous calibration process after the model is launched.
【 Reference Source 】 Comprehensive compilation of industry information released publicly